Partnership Projects: A Framework for Evaluating Public-Private Housing and Development Efforts

Published by the Government Accountability Office on 1990-05-22.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                                                                                         United States
%-A0                                                                                     General Accounting Office
                                                                                         Washington, D.C. 20648


                                                                                         May 22,199O.

                                                                                         The Honorable Henry B. Gonzalez
                                                                                         Chairman, S,ubcommittee on Housing
                                                                                           and Community Development
                                                                                         Committee on Banking, Finance, and
                                                                                           Urban Affairs
                                                                                         House of Representatives

                                                                                         Dear Mr. Chairman:                 ,:,
                                                         .:/   ,‘!               ,:        ‘.     :r:’
                                                                                                               :   .;.T’.

                 _,                     ,;
                                                         ,,’ _:_                        ;IThis report was prepared inresp’onse to your request for a framework
    ; .,, ,.;                          ‘,                                                 to ev,aluate the. success :of’.public-private partnerships in housing and
                                                                                          &mmunity~ development.          ;             .

B&&g~$)und.,                                 .i.                     ;:          ‘.‘.     A~,partnership
                                                                                                     ,‘, i _,,,.: in.Chicago workswith                   local lenders and neighborhood
          ., .,                   ,‘., ‘., ., “_                                    ., . prg~~~z,a~ions,t?,~~s~~~~,~,neighborhood-based projects that’ rehabilitate
            /            :,                    ,‘,,S                  ,_.I               rhousing.
                                                                                              .‘.,d._. ,for: loJw@ome tenants; A -partnership of local residents, the
   .‘Y~,                             : ,,: :                ,(.’                  ‘,a. xitygpvernment,.                    finan$e and ,development professionals, and area
,!. ).I.,.. ; _, ..‘I . . 3 ,                                  :- .. .;,, :: churches@ Baltimore,,orga$zes,a:series                                         of construction projects aimed
                   ‘, 1                                                   i.
                                                                          ( ! ./‘,I at revitalizing                  a commercial .area, iHave these and other such partner-
                                     ,’ 0 .j,: /                                               ,,,, ).(I
          ’; ,.1 !-1                                           /.a.I/ N( ,ship 8projects been ,s,uccessful? Did the Chicago partnership succeed in
                                                                                          meeting. the needs ~$low~in~orne..residents in the affected neighbor-
                                                                                          hoods? Even if the economic health of the Baltimore commercial district
                   ,(         ,,/.,II                          ,,.,.1., ,, : : im?rpved, wouldthe, areahave developed as well or better in the
            :,_.,              ,,        i-,,yI : ,.,.                       ,, .., absence of the;partnershipT             ..,.N             il    :, .
    II’              :‘,                    .;             , ‘:”               ,,   I2    *  ...I  %,, .‘,:I /:I”  jib’‘,,     !i     (I,..,.
                                                                                                                                            I    ,,.
                , : ,,/                .I .;         : ;! ”1 ‘i.‘!,. :: :,y :,,Al~!p~~gh,publicrpriv~~~~pa~,~ersh~ps                                      appear to be a popular way of
                                          3, :,,                                     : !,addressing, &al,               ,housing: and ,Fomnurnity development needs, claims of
            ,‘i,,+   ,’                                    1..
                                                                     ,,I:      ,’

                ‘_ /.                   1, <I ,,I’ ,‘I,        6                  ‘j ,their s,uc~~essas aZpoli~y,tool have been based largely on anecdotal evi-
        .,’                                  r,       ,.   ~ ~ .I  ,    /
                                                           ,,? 1.1 ,,I>~~nce.l,Ee~,~~~empts~?h~~e,,be;e~~nade to validate these claims systemati-
            - :’            ,‘I     8:     /S” :j:, ,, ‘, jv.,-,J;,,,tally, This, frame!work!                    ?..,.11’         isl,designed to fill this gap by providing a
              _. , :,) ( >,’ ‘.:* ,                            * ; fij I ‘comprehensive,,l,, set                         1-.:of criteria ,ag,ainst which to measure the performance
                                                                                        :of. these . :’  organizations.           in specific projects and across federal programs,

                                                                      We defined public-private partnerships as joint efforts between the pub-
                                                                      lit-sector and. ,either the private for-profit sector or the pri.vate’nonprofit
                         IL                            :I_            sector. In Contrast to priv,atiz-ation, contracting out, &other arrange-
     .‘,,             ,:,’                                 ,’         ments . between the I@+ andiprivate
                                                                                                      ..,         sectors, a partnership signifies
                             ,, ,,.’                      * ,:        that both ,public and .priv,ate sectors share risks and responsibilities in
        ,.,           ‘,. . ”                      .,,     ,,  _’ ,,order t.omeet critic,@ community, needs, as defined by the partners.
                                                               ; I,. ,,Shared, riskmexnsi that ‘both .partners could lose resources; it encourages

                                                   .‘,                      ,’
                                                                                        ;Page 1                                   GAO/PEMDBO-9   Partnership      Projects   hamework
                                                      B-236100                                                                                   r’

                                                     .,the involvement of both public and private sectors in ventures that
                                                      neither sector could successfully attempt alone. Shared responsibilities
                                                      include joint decisionmaking by representatives of the different groups
                                                       who work collaboratively on the project.

                                                      sets of criteria focused at the project level: the needs that a partnership
                               ,.                     project addresses, the process by which the partnership operates, and
                                                      the’ outcomes of the project. We examine how to apply these criteria to
                                                      individual local projects. We then move to the issue of how to evaluate
                                                      federal,support for such projects. (An overview of the framework is pro-
                                                      vided in appendix’I1. ‘See.table II. 1.)

                                                                                                       .a.     1
          Need- Crit,er?ia-                          “I The need for a project can be considered in terms of themagnitude of
-                                                       I r@ed,‘definedas the difference’between a standard of what ought to ‘be
                                                           and existing-conditions. For example, rents in an area could be com-
     -I                                                ,, pared ‘to’s common~standard for affordability, such as 30 percent of the
                                                           household income of tenants,; as a measure of the need for more afforda-
                                              ,            ble housing. If rents were’ found to ‘exceedthis standard, that could indi-
             *                             ,.              cate,that -a’project! to build more low- or moderate-income housing in the
                                                $,         area is-justified:              ”
                                                   ;Qi:,, .’ ‘:           ,”      .:         (,
                    1                                  ” This suggests another aspect of need: distribution of housing or commu-
                                                           nity development needs. Thepartnership may identify a geographic
                        ,‘,   .,         .                 area or target population to be served by the project. A task of the pro-
                                                          jecf ‘may th~~~‘lj~‘tro’~~tch’th~‘~e~~ices   it provides to the needs of the
                                                  .,       target area  or   ‘population: When  effectively  done, a partnership project
                                   :   ’                   aimed at providing housing ‘to a’low-income population in a mixed-
                                                         ’ income neighborhood, for example, will identify and select as tenants
                                                           low4ncome’households out‘of thatmixed population. (Details of the cri-
                                                          teria that apply to the need dimension and the associated indicators and
                                                          measures are,discussed in appendix III.)

          Process Criteria                       ‘“* Process criteria deal with the implementation of a project and include
                                                     planning and’initiation of the project, the structure of the partnership,
                                                     management of partnership operations, and resource acquisition and
                                                     management; understanding how a project was implemented can reveal
                                                     important information about why the project succeededor failed and
                                                     could identify ways in which programs can be improved. For example, if

                                                     Page 2                                 GAO/PEMD-90-9   Partnership   Projects   Framework

       \                                                                                                                          B-236100


                                                                 ” nomic development prbject had conflicting and incomp
                                                              Yr‘-‘Yr$$t’e’$&in              ‘what went wrong ‘and suggest strategies for more success-
                                                                      ‘ful implementation of future projects. (Details are discussed in appendix
                                                                                                                          ,.,,;,-,,..& ,?1,.: ,,. IS,“.‘, ..,
                                                                       ,                           ,.                                     .       i ’
           OtitComk @x+ria                                              Outcome criteria ‘relate to the effects of a given project, including tangi-
                                                                        ble effects, such as the’ number of housing units built or amount of com-
                                                                     8mercial sPacedeveloped, and less tangible effects, such as changes in
                                            ,_                ‘,’ the environment for investment ‘in a community. Outcome criteria refer
                                                                     ,!to how well a project’fulfills the housing or community development,
                                   ‘, ,,’        ‘,,      “b /; needs it is intended to ‘address, how it affects the public sector, commu-
                       ‘,                     ,’                      ’ nity residents, the private sector; and the partnership organization
  ;1                                                                    itself, and how much it costs in financial, political, or social terms. Thus,
.1.                                                I
                                                                        if we found ,that a given partnership project produced more low- income
                              “,’                                : )‘housing in a target neighborhood than would have beenbuilt in.the ‘,.
                                                                  ” absence of the partnership, ,we could conclude that the project had been
                                        /                               a success;(Details are discussed in appendix V.)
                                 ; “,,                           8.: ’                  .,:           ‘f/      ,,I ‘,
                ,j >I.,. 2: ‘,. ;’ ,* ,                              ,               ‘,                 :,.       ‘,
           lZ’Vti~u~ti& Ffqm the                                        We’turnnow from the local or project level to considering how to evalu-
           Fedei?l, p.rsL4’4ti4e                           :’ L‘. ” atea group or’prograin of federally assisted public-private I#tnerships
                                 ,                                      in housing and community develobment. T
                                                                        gram      v&h&hegdi.r&A~obj~f~~
               :‘,            .,,,.. ‘/                                ‘community~development~~artnerships. However, we raise four major
                      1 .:;.;,: ,,‘:‘,’     -,,):I. .,. ?,;\, ‘queStions that can be used to,guide the evaluation of the set of partner-
                   ; I : ,, i              ; : ,’                    I ship projects’funcled under a harticular program (such as the rental                                                                                                                            I
                  * i, “f’,, ,<I:                 ,. ,a,;       ! .:**:~odsi~~,rehabili~sitidnprogram) or to do comparative analyses of part-
                    :      : ,,, ,      :                       : ‘I nership~projectsacross programs~(such as all federal programs that
                                                              ‘,,              :,          ‘, :i:    ( !i..,

                              5:,, ;          “_,‘:,                    a&&rental housing ‘construction). These questions are
                       !,, I ,! ,/,,I li*$!, ,,(            ,                     > ‘; . I_                           ,’
                     ,,.’ $Ji’, .,I                           I,,
                                          (/.I,., r,i., ..: 1. Whatfederal resources’are allocated to support public-private part-
                        ,, j ‘,‘, .: ,, ;,, 1’                    j nership.projects in housing and’community development?
                                          ,:’                             ::,             ,I                  ,‘!, :‘.,
                                                                        2. What needs are addressed by federally assisted public-private
                             :.                                       -pafinership&,                        .        ~-,,            .
                                                         ,'                                         ,,           !.       /.",,                                              ,,   ,.                                                       '/
                ,,',,I        'I            ,   ,                              (_I                              ':,
                                                                                                                                  .3. How well is the implementation of federally assisted partnershi;’
                              ‘..,,,. : ,:                                     ,;-_ i                                       f' pfojects               m&itored?              "'                J       '*'         _I
                                   'jf-".           ,;               ',                    .I                                                             ;     ,', "   ,,             'i.'.           ;,

                                   .:."              ..,                  !/                                     .,
                                                                                                                            ” 4. How successful
                                                                                                                                       .., ,, ‘arefederally assisted partnership projects?
                                                                                                         , .:,                ‘.:‘_ ;

                         ‘>        ,:                          ,,;        2,         ‘.’             ,,::             .      /,
                                                                                                                                  Page 3                                                           :         ‘GAO/PEMD-90-9   hrtnership        Projects   Framework

                                              ---Aswe ha~,~~~~e~~~~~p~~~~~~~~~~~~~ects:                   Federal Sup-
                                                pG%f&Public-Private Housing and Development Efforts (GAO/
                                                /tie                                 information is available to answer
                                                these.qu%%R@Ri~&al                programs that support public-private
                                                partnership projects. In this framework, we identify some of the key
i                                               indicators (such as the number of partnership projects supported and
                                                the amount of private funding leveraged through federal support) on
                                                which information could be collected and maintained by federal agencies
-I                                              for purposes of evaluating the partnership projects they fund. (Details
                                                are discussed in appendix VI.)

     Applicability of the                       Ideally, the framework should be useful as a guide to evaluating the
                                                need for, implementation of, and ‘outcomes of housing and community
     Framework,                                 development projects undertaken by public-private partnerships. (See
                                                table II. 1.) The fact that the framework is very broad does not require
                                                an evaluator to use all the criteria, however. An evaluation may focus
                                                entirely on outcome criteria, for example.
,                                                         Setting forth evaluation criteria implies the need for measuring perform-
                                                          ance against those criteria. Therefore, for each criterion we present one
     ,                                                    or more indicators that evaluators can use to assessthe extent to which
                                                          a given project or set of projects meets the criterion. For each such indi-
          <.I                        . .111‘1.’           cator, in turn, we present one or more specific measures. For example,
                      - /I:jl.,,.., J/,* ,.,, / ,.,,I , 1,I:.
                                                                 <*,I >.r,.m       need,for a housing project is the magnitude of hous-
                :    I(                                   ing need. (Seetable 111.1.)One indicator of this need is the extent to
                                                          which housing:in, an area is not affordable. To,measure the degree of
                                                          affordability, the evaluator might consider the proportion of household
                                                          income.going for.rent, the rates of homeownership in the area, interest
                                                          rates for home mortgages, or the ratio of shelter beds to the homeless
                                                          population. The&precisechoice of measures would of course depend on
                                                          the relevance of the measures for the area being considered for the pro-
                                                         ject and the.purposes of.,the,project itself (for example, whether it
                                                          involves constructing rental housing or owner-occupied dwelling for
                                                          families or single room occupancy facilities for individuals).

                                                The measures we present vary considerably in terms of the availability
                                                of reliable data. Some rely on data such as census reports, which may
                                                become outdated, while others-especially those concerned with project
                                                implementation-require more qualitative or impressionistic informa-
                                                tion. Evaluators would need to make assessmentsof the extent to which
                                                specific analyses were needed or feasible, given the scope of the project
                                                and the resources available, before proceeding.

                ..                              Page 4                                   GAO/PEMD-90-9   Partnership   Projects   Framework


                               The.framework is not-onlya model for looking at past performance,
                               however. It can also serve as a planning tool for federal, state, or local
                               officials and private-sector participants in public-private partnerships.
                               Used prospectively, the framework could identify the information that
                               will be needed to evaluate the project at various stages of development
                               and could clarify the information gaps that may be too costly to fill.

                               We discuss the scope of our ,work and the methodology we used to
De’velocment of the            develop the framework in appendix II.

Agency Comments                 We received comments on this report from the Department of Housing
                                 and Urban Development (HUD). (See appendix VIII.) Those comments
                                describe the report as ,“solid” and “well-written” and express agreement
                                with our characterization of the difficulties of carrying out the evalua-
                                tions discussed here because of the problems resulting,from ‘%helack of,,
                                readily available, reliable data and the high costs associated with col-
                                lecting the needed data.” They also note that the report could be useful
                                for improved monitoring of projects or for providing technical assis-
                              , tance to partnerships. Finally, HUY proposes several steps we could take
                                to .encourage the use of the framework.
                                Copies of this report will be sent to the Subcommittee on Housing and
                                Urban Affairs’ofthe Senate Committee on Banking, Housing, and Urban
                                Affairs, the Senate Committee on Governmental Affairs, the House Com-
                                mittee on Government Operations, and the Secretary of the Department
                                of <Housingand Urban Development. In addition, we will make copies
                                available to others upon-request.:‘.
                                         ,’ ‘,           ‘)I
                                If you have any questions or would like additional information, please.
                                call me at (202) 275-1854 or Kwai-Cheung Chan, Director of Program
                                Evaluation in Human Services Areas, at (202) 275-1370. Other major
                 -,   :         contributors to this report are listed in appendix IX.


                               Eleanor Chelimsky
                               Assistant Comptroller General

           I.9            /
                               Page 6                             GAO/PEMD-90-9   Partnership   Projects   Framework

 c -   .--   .-.. --_   .-.                                                                -.--._.      .- .   .


               Letter                                                                                                                                                                                             1

               Appendix I                                                                                                                                                                                 ”       10
               I&c.j~est Lettep
              Appendix II                                                                                                                                       ,’.i                                              11
              Overview of the                                                              Evaluation Criteria                                                                                                    11
                                                                                           Application of the Framework                                                                                           12
              Framework                                                                    Objectives, Scope, and Methodology                                       ~                                             13

              Appendix    111                                                                                                                               I.:,, A.’ ,,.~!
                    ..,                                                                                                                                                                              ,,
                                                                                                                                                                                                          ““., 17
              Criteria’of Need                                                             Steps in Needs Assessment                                                                                          17
 h                                                                                         Problem Magnitude                                                                                                  19
-1                1.’                   ,,               ,’      .,,.         ,.           Duplication and Appropriateness                                                                                    27
              Appen&xI$T                                                ,;::                    ,‘,   ,‘:          ,.,)       ..             ‘_
              Pro&&                     Cpiteria                                       : ,‘;lEz$ df& ;;tiner;;ip.                                                                                             31
                                            ”                                                                                                                                                                 34
                                   ,,        .,,’                                      :   Management of Partnership Operations                                                                               35
                                                                                           Resource’Ac&isition and Management                                          -.                                     39
                                             I, ”                        ..                Application ,of Process Criteria                                                                                   41
               ..I                                                                              :
              ~                                                                                   ‘..,,T,f’ ,I rj: ..
                                                                                                            ,.        ..r                                                                                     43
              outcome            @i&&a                                            ’        Design Issues                 _’,.,,, :,                                                                           43
              ,, .’ .“;i ‘: ‘/,.ijl ,. ,!’                                    i ,,,,       Achievement of/Intended, Objectives                                                                                45
                                                                                           ()t,herEff&,si:,               :        ,.,   r        ’ #I                                                        48
                                                                                           Costs of the Partnership Project,,,,                                                                               53
                                                                      I’ ;                           ‘,,    .,
              &jjp&                            VI>             ” “;:‘:,m 1:                ,.          .y i    ;, .,;
              EV&ating l?ubli&                                                             What Federal Resources‘Are Allocated to Support Public-                                                            58
                                                                                              Private PartnershipProjects in Housing and
              Private Partnerships                                                            Community Development?.
              From a Federal                                                               What Needs Are,Addressed by Federally Assisted Public-                                                             60
              Perspective                                                                     Private Partnerships?
                                                                                           How Well Is Implementation of Federally Assisted                                                                   61
                                                                                              Partnership Projects Monitored?
                                                                                           How Successful Are Federally Assisted Partnership                                                                  64

                              “’   ”    .:    :,   ,:,    ,,             .,
                                                                                           Page 6                                                        GAO/PEMD-90-9      Partnership   Projects    Framework

Appendix VII                                                                                                    “67
Expert Reviewers
Appendix VIII                                                                                                     68
Agency Comments
Appendix IX                                                                                                       70
‘Major Contributors to
This -Report             ‘.
                                                                                            :              (,
Bibliography                                                                                                      71

Tables                   Table II. 1: Overview of the Evaluation Framework for                                    11
                              Public-Private Partnership Projects
                         Table III. 1: Need Criteria: Problem Magnitude                                           20
                         Table 111.2:Need Criteria: Duplication and                                               27
                         Table IV. 1: Process Criteria: Planning                                                  31
                         Table IV.2 Process Criteria: The Structure of a                                          34
                         Table IV.3: Process Criteria: Management of                            ‘..               36
                              Partnership Operations
                         Table IV.4: Process Criteria: Resource Acquisition and                                 39
                         Table V. 1: ‘Outcome Criteria: Achievement of Intended                                 46
                         Table V-2: Outcomes: Other Effects                                                     49
                         Table V.3: Outcomes: Costs of the Partnership Project                                  63
                         Table VI.1: Federal Support for Public-Private                                         58
                         Table VI.2: Need for Public-Private Partnerships in                                    60
                             ,Housing and Community Development
                         Table VI.3: Monitoring the Implementation of Federally                                 62
                             Assisted Partnerships
                         Table VI.4: Program SuccessThrough Public-Private                                      65

         ,,.‘_      I    Page 7                           GAO/PEMD-90-9   Partnership   Projects      Framework





                                                                                                                                                                              ,’            .                 .

-=                    1   “’               ‘,        i   :             I    ,I,


                                                                                                                                                               :         I)//,,



                                                                                                                                                                                                                          (,’                 ”
                                                                                                                ‘,          \,.                                                                                                                          ;

                                   ,:..’                                          ,:           )/,                     ,’
                                                             S!’                                                                                           I             ,’                                               .,,                      ‘,

                                                                       *,                            _,,
                                                                                                                                                                                   i                         :“‘,,                                           :


                                                                                                                                                                                                             .,I                     i,

                                                                   ?,,‘,                                            .‘.,                    ,:’                                                 ,,                              .




                                                                                  AHS                                                  American Housing Survey
                                                                                  GAO                                                  U.S. General Accounting Office
                                                                                  HUD                                                  U.S. Department of Housing and Urban Development

 rL. ,, ,   ,)I, ;/                                                         : ;   Page 8                                                                                                                                                                     GAO/PEMD-90-9   Partnership   Projects   Framework

                                                                                                                                               .-,                   ‘.

                                                                                                                                :.         /

    (‘.                                              .,

                                            :              ;:                      ..,‘.”                      _,           ’

                         .’             ”
          .,         r                          **                   -..                                                             .,,                   _.                  \’

               .;^            ;.                                                                          .,




                                   :.                     ,.,                      . .                              ,                            ,’

                                                                ,_         ,.               ,,i   ‘,.‘,


                                                                                                                                                                ,’                  1

                                                                                Page 9                                                                          7 GAO/PEMD-90-S         Partnership   Projects   Framework
i    Appendix .I                                                                                                                     n

     Request Letter

                                             U.S. HOUSE OF REPRESENTATIVES
                                              SUBCOMMITTEE    ON HOUSING    AND COMMUNITY
                                                                  OF THE
                                          COMMITTEE   ON BANKING,   FINANCE AND URBAN AFFAIRS

                                                        OiE HUNDRED FIRST CONGRESS
                                                      2129 RIVBURH HOUSE O~ICE BVILOINO
                                                      WASHINGTON,    DC 20616-6062

                                                                May 15, 1989

                      Honorable Charles A. Bowsher
-                     Comptroller   General
 1                    v. s. General Accounting
                      441 G Street,
                                      N. W.
                                    D. C. 20548

                      Dear Mr. Bowsher:
i                         The House Subcommittee on Houtiing and Community Development is
                    interested    in whether public-private      partnerships   are successful    in
                    meeting the goa+s,of federal housing,,and community development
       ,‘,         ‘progranis:.   We under&&d    that the,"Prognam Evaluation       and.Nethodology
                    Division    of the General Accounting.Office       is developing   methods for
                    performing    evaluations  of such part&rships.
                          .-,In connection with our work on the Housing and Community
                      Development Act of 1989, we are interested        in having information  on
                      the success of public-private    partnerships     in a wide range of housing
                      and community development programs.       Specifically,    we would like your
                      staff    to:
                           1.    provide descriptive            info&ation    on federal programs
                                 currently    supporting         partnerships   in housing and community
                                 development;    and
                           2.    develop a framework            for    evaluating         the success     of such
                          The staff    qf the Subcommittee would like to meet with your staff
                      to discuss details     of the study~and reporting schedules.   If you have
                      any questions,~please     call Gerald R..McMurray at 225-7054.


                          Pirge 10                                            GAO/PEMD-90-9     Partnership   Projects   Framework
Appendix II

&ervieiv of’the F’ramework                                                                                                                                                                            ,:

                                                                     ,,In this, appendix, twe give an overview of the evaluation framework and
                                                                       discuss the methods we used to develop it.

Evaluation Criteria                                                    The framework consists of nine criteria organized into three categories:
                                                                       the needs that the partnership addresses, the process by which the part-
                                                                       nership is implemented and managed, and the outcomes of the partner-
                                                                       ship project. (See table II. 1.) The first criterion under need-problem
                                                                       magnitude-focuses on the size and distribution of housing and commu-
                                                                       nity .development problems. The second criterion, duplication and
                                                                       appropriateness, is concerned with determining the efforts already
                                                                       under way for addressing the need as well as the appropriateness of a
                                                                       partnership project relative to those other ways of addressing the need.
                                                                       Process criteria include the general management issues of planning and
                                                                       resource acquisition as well as issues that are uniquely important in
                                                                       partnership ventures2that is, the structure of the partnership and the
                                                                       management of the partnership. The last three criteria address the out-
                                                                       comes of a partnership project: whether the project has achieved its
                                                                       intended objectives, whether the program has had other unintended or
                                                      :’               secondary effects, a&what the costs of the project have been.
Table ll,.l: Overyieyof ttiq Ev$uatioq
Framekork’for Public-Private                                         ‘: ,&teggy                                 .:           ‘.                   Criterion
Partnershjp Projects                                                   Ne$d fo! the partnership                project        ’ :,                Problem magnitude
                                                                                                                                                  Duolication and aotxooriateness
                                                                                                                                                                    . .  .
                                                ,(,             .,
                                                                       p&e&       bf par!ne’rtihip           project              ”       “’      Plan,ning
                                                                       !,yplementation                                                            Structure of partnership
                                                                                                                                                  Management    of partnership operations
                                                                                                                                                  Resource acquisition and management
                             :t.        ‘,
                                           1.,                         Outdomes           of bartnership’project                                  Achievement    of intended objectives
                                   / 1 ” ..‘. i’,          ,,                                 .1      ., ,;“I                         (
                                         * .,
                                                                                                                                                  Other unintended     or secondarv I effects
                                                .,.                     '_     :   ',.'           )     ,I

                                                                    The nine,criteria were developed to categorize the types of issues raised
                                       ,,a                          in evaluating public-private   partnerships and the projects they imple-
                                    r.,                                             :.,
                                                                    ment. For each criterion; relevant indicators are identified and measures

                                                                   ‘are suggested. This s,chemeis not the only categorization scheme possi-
                       8,      _”                                   ble, nor do these criteria incorporate all the issues that could be raised
                                                                    about public-private partnership projects. Instead, the framework pro-
                                                                I”’ vides,a heuristic inventory of indicators and measures that are relevant
                                                                    to the evaluation of public-private partnership projects in housing and
                                                                    community development.

                                                                     . Page 11                                                                 GAO/PEMD-90-9     Partnership   Projects   Framework
                                                             Appendix   II
                                                             Overview   of the Framework

                                                             The framework outlines the kinds of questions that are appropriate in
i   Application of the                                       assessing the need for, implementation of, and outcomes of local part-
    Framework                                                nership projects. Not all the measures and analyses included in the
                                                             framework have to be used in every evaluation. Our intent here is to be
                                                             comprehensive; but only appropriate criteria and measures need be used
                                                             in any specific application. For example, an evaluation of the implemen-
                                                             tation of partnerships under a given program need not take account of
                                                             measures of outcomes or,of need for the partnership.

                                                            In addition, the suggested measures vary widely in the extent to which
                                                            data are’likely to be available,,eitther through extant sources (such as
                                                            census reports) or through originaldata collection (including surveys

                                                            and observationaltechniques).       Throughout the report, we address this
                                                            feasibility issue for individual suggested measures.
                                                                     ; .,                          ,’
                                                            Several steps are needed to use the framework to evaluate a project or
                                                            program. The first is to decide:the purpose and scope of the evaluation.
                                                            For example, to assess the implementation of a project, an evaluator
                                                            would focus on’the process driteria’and indicators but probably would
                                                            not, deal With partnership outcome issues at all.
                              ,,             .’
                                                      I,      Once the purpose and. scope of the evaluation have been decided, the
                                                              second step is deciding on sources,of information and collecting data. “’
                                                              Generally, information on eabh criterion should be drawn’from as wide a
                                                              set of sources as possible and should be reviewed for its relevance and
                                                           ‘, methodological quality. The final steps include assessing the quality of
                                                  1           the data and synthesizing information from different sources and on dif-
                ,..’                     ,
                                   ”                          ferent measures. .It will be necessary to set priorities and to decide if
                   ,.                                         some information’may     be too &stly to collect. Answering some of the
                                                                    ./ ,. posed may  be prohibitively expensive.

                                                              As noted below,, documentation on partnership projects tends to be pro-
                I                  i,.                        motfonal. This,,suggests, that there may be a difference between the evi-
                        .,.                                ,. dence that is <available and what ,actually happened. Even if partnership
                                                              operations and effects have been accurately documented, some data
                                                              may still be’ difficult to obtain. For example, the private sector may be
                                                              reluctant to reveal sensitive data ,on financing and development costs or
                                                              proje& performance. In addition, because a project’s success reflects on
                                                              both public and private sectors, it may be difficult to obtain information
                                                              on partnership projects that have not met expectations. Other informa-
                                                              tion such as how the partnership’ was initiated and project activities
                                                              negotiated may not be revealed through the normal documents and
                                                              records that an organization might keep.

                                                            Page 12                             .GAO/PEMD-90-9   Partnership   Projects   Framework
                                                                        Appendix   II
                                                                        Overview   of the Framework

                                                                        .These three concerns-the     validity of available data, the accessibility of
                                                                        data, and the lack of data-make       it probable that the evaluation of
                                                                        partnership projects will require the collection of new data through
                                                                        surveys, interviews, and observations rather than relying only on
                                                                        existing records. Again, the user will have to decide from the available
                                                                        budget what is feasible in terms of cost. In the explanation of the frame-
                                                                        work that follows,, we identify some potential sources of information for
                                                                        the measures that we have indicated.

                                                                              In short, users of the framework need to make additional decisions
                                                                              about evaluation design, the relevance of specific indicators, and the
                ‘.    .’
                                                                              feasibility of collecting data on suggested measures and analyses. In
                                                                            ‘additionto guiding the evaluation of specific partnership projects, the
                                                                             framework can facilitate the comparison of data across projects by pro-
                                                                             viding a common set of criteria for categorizing data. The framework
                                                (.                           may also be useful in the development and design of partnership
                                                                             projects, because it suggests measures for assessing the need for a part-
                                                                             nership project, implementation factors that may be related to project
                                                                             success, and the outcomes or effects of the project.
       1               I /:                          :                              ‘,
                 /i   :, ; ‘ :                 $;:).:       “,.. ,,                            ‘.
                                                                     .’ ” We defined public-private partnerships as joint efforts between the pub-
   ., ._,  @Cope,iid                                          ~..,           lit, sector and either the private for-profit sector or the private nonprofit
Methodolog$                                                                  sector. In contrast to privatization, contracting out, or other arrange-
                                      _j              :                      ments between the public, and private sectors, a partnership signifies
                      ,,           ,/‘.                                      that both:public.and private sectors share risks and responsibilities in
                                                                             orderto meet critical community needs, as defined by the partners.
                                                                             Shared risk means that both partners could lose resources; it encourages
      ,‘,!’           ),‘        ‘;L                                        the involvement,of both public and private sectors in ventures that
               ! !’ .,s, .:                                                 neither sector could successfully attempt alone. Shared responsibilities
     “/I”’ ,.I       h,,      ”                           1,.               include joint decisionmaking by representatives of the different groups
    ,.            .,      I,,,‘:!)..                                        who work collaboratively on, the project.
 *_.       ,.       / I    ‘.                              ’                                      : / 1;
   ?_                     ,.,.a!                                ,,, I Although,partnerships              are found in many policy areas (including job
                                                                            training and education), we restricted the application of this framework
                                                  1                         to partnership projects that focus on housing and community develop-
                                                                            ment. Housing may ,include construction, rehabilitation, rental assis-
                                                                            tance, and other activities.. Community development refers specifically
           ‘,                                                               to efforts directed toward neighborhood revitalization, economic devel-
                                                                            opment, *and improved community facilities. Our definition of commu-
                                                                       1I i nity development excludes projects that focus solely on community
                                    ,!I                             ‘, organizing, job training;and, other community services.
                            ,,‘.                ;,                     .,,,            ,’ ‘I             .’

  ;                        .I,,           (,                .//’   ,.
                                                                        Page 13                              GAO/PEMD-90-9   Partnership   Projects   Framework
                                                           Appendix   II                                .,
                                                           Overview   of the Framework

I                                       ;                                           : Partnerships can vary according. to purpose and duration. One type is
                                                                                       the project-based partnership, which is not permanent, does not consti-
                                                  <                        ,;        ctute a.formal.delivery           system, and may not lead to another venture in
                                                                  ..;                  the    future   (U.S.     General   Accounting Office, September 1989). A second
                                 ,.,                  ‘.                     L’ type is program-based and includes, both the public and private sectors
                                                  I ‘. _’                              as participants, has access to resources, is ongoing, and tends to be more
                   5:                    ./           ‘,                               formal than project-based partnerships. Project-based partnerships tend
                                                          I.(,’                       .to  be single project, partnerships while those that are program-based
                             ., ,,-                                                    tend to be multiple project partnerships. Evaluation methods and
                                  I’,. :                            .’ ,, ‘, requirements for these two, types,may vary.
                         : ,,,’                  “’ : :                            ”
                       ‘:.I,                 i’ . .._.                         ._: We addressed both housing and community development with one
      ,, ..,               ,‘. .,, I ,,                              _I                framework because we found considerable overlap between housing and
                                                                                       community development projects. Many community development
                                                                                       projects involve housing activities as well as economic or infrastructure
                   ~                  ‘:          ;,,, “_j                             development. For example, the Inner Harbor project in Baltimore con-
       ,.;           ;.          ,.: ,I ,                     t ,,,                    strutted mixed income housing units, as well as assisting commercial
              I “..,.                 “.                                               development..
           ;.                                                                            ..,,,’               ‘.
        ‘.                                                                           ‘Building on the general evaluation criteria developed in Children’s Pro-
         .               >                                         ”                 ,grams: A Comparative Evaluation. Framework and Five. Illustrations
       ‘.. 1                   *.,;,y.                                    3 .,: (GAO/PEMD+~-?~BR, August 1.988), we identified relevant indicators and
        ,’ .,                 j / ,. )                      I : +.                     measures for ,evaluating partnership projects in housing and community
                              , .’             ,I          ‘., ;:                      developme,nt. ,The development a@ assessment of the evaluation frame-
              / ; ,‘_’                         ,,’ i ,” .‘,                           workinvolved, four steps: (1) literature review, (2) development of the
                ./                                                                     framework, (3) expert review of .a,.draft framework, and (4) revision of
                                                                                      the framework based on further research.
            ..)’ ,, 1,:,, , ‘j,,I I                ;                ‘.         ‘. : ,;               .,,:2’ :      )     , , ‘, ‘: : ”
            ,, > ,, ,, :                   )‘.                            .: ;,. We reviewed studies and reports on public-private partnership projects
    ,’ /, ,’I,” ;..:. ,.                                                          :’ and .othes housing and communitydevelopment                  projects in order to
        ::*               ‘. ..‘,                .‘:          *.,                ‘, develop relevant criteria, indicators, and measures for evaluating part-
                      .) ,, :                  :                                      nership.projects. .(A bibliography, of the materials we reviewed appears
                                                                                      at the end of this report.) We found that the literature on public-private
                                                                                      partnerships tends to promotei ,rather than evaluate, partnerships. For
                               ;_ ‘,’                     ,‘, .,‘,                 ! .example,. SRI, International published several reports under contract to
                      :, :                                         .,                 the U.S., Department of Housing and Urban Development that were
                                          ,.,,               .‘/.                     intended to guide, local government,, local firms, and corporate involve-
          ,.’                                       i.                  I             ment in public-private .partnerships, But despite the fact that they were
                   :            ,. ,,’                                          .’ more. promotional than evaluative, these and other “how to” guides
      .’ 1,                                                                          proved useful in identifying process variables. Because they were
           ,,’          ‘( .:;                         ,.             ., :“. .intended to encourage partnerships, they emphasized “keys to suc-
                                              ,. ‘.                         .        cess” -elements-or            variables that are important to consider in the initia-
                      :                   ~,                           .,            tion, planning? and implementation. of a partnership project.

         ~’                ,.     .‘:       3.   ; :. .’   Page14                                        :    GAO/PEMD-90-9        Partnership    Projects   Framework
                              Appendix   II
                              Overview   of the Framework


                              While the promotional nature of the literature on public-private partner-
                              ships facilitated the ‘identification of process variables relevant to the
                              evaluation of partnership projects,‘the literature was less useful in the
                              objective identification of need measures. The needs to which partner-
    i                         ship projects are addressed tend to be described in dramatic rather than
                              operationally defined terms. This example is typical: “In the 1970s Old
                              San Juan, the city’s historic core, was obviously headed downhill.
                              Residents were moving to more affluent neighborhoods, buildings were
                              deteriorating, and the area had become known for scarce parking, sleazy
                         _’   bars, and drifters.” (U.S. Department of Housing and Urban Develop-
                              ment, March 1987) While need measures such as migration, the physical
                              quality of structures, and the availability of parking are implied in this
                              statement, it is not clear which measures were used or how the need was
                              determined by the partnership.

                              Because need tends to be stated in general terms, outcomes are not
                              directly linked to these needs in the descriptions of successful partner-
                              ships. Instead, the literature on partnerships emphasizes tangible out-
                              comes such as the number of housing units constructed or rehabilitated,
                              the number of jobs created, or the amount of money leveraged. In addi-
                              tion, given the promotional nature of the literature, it was difficult to
                              find discussions of failed partnerships or negative side-effects of part-
                              nership ventures. In order to gather more information on relevant vari-
                              ables in assessingneeds and outcomes, we relied on evaluations of
                              nonpartnership projects and programs in housing and community

                              The literature review not only provided criteria, indicators, and mea-
                              sures but also enabled’us to identify issues in evaluating partnerships.
                              For example, Lipman discusses the complexity of the leveraging ratio, a
                              commonly mentioned measure of success in obtaining financial
                              resources. (Lipman, 1988) We discussed this and other issues in evaluat-
                              ing partnerships in relation to specific measures.

                              The draft framework that we developed from the literature was
                              reviewed and assessedfor comprehensiveness by an expert panel (listed
                              in appendix VII). We sent the draft framework to the panelists and
                              asked them to first generate their own criteria, indicators, and measures
                              for evaluating partnerships and then to review and comment on those
                              that we had developed from the literature. We then brought the panel-
                              ists together for a day-long meeting to discuss the evaluation of public-
                              private partnerships in general and the contents of the framework in
                              particular. The comments of the panelists were incorporated into the

        8   i   I   .:        Pirge16                            GAO/PEMD-90-9   Partnership   Projects   Frayework

                Appendix   II
                Overview   of the Framework

                framework where appropriate. The framework          was further       refined and
                sent to. the panel members for a final review.




     ,,i.   :   Page 16                          , GAO/PEMD-90-9   Partnership   Projects   Framework
       Appendix III

       ‘Criteria of Need


                                                                    An important issue in evaluating any housing or community develop-
                                                                    ment proje.ct is the need to which the project is responding. The purpose
                                                                    for assessing need is to provide information to the planning process to
                                                                    enable the prioritization of problems and the selection of appropriate
                                                                    activitiesto address them. Evaluative data on the status of the need to

--:I                                                                which ,a project is responding operate as a baseline against which data
                                                                    on project processes and outcomes can be compared. Without an under-
                                                                    standing of the nature and extent of the housing or community develop-
                                                                    ment need;it is difficult to evaluate the appropriateness or success of
                                                                    the a&ions taken or the outcomes achieved.
                                                            .   .

                                                                                  , The extent to ,which a needs assessment is appropriate for any given
                                                                                      I1project“depends in part on’ the. size. and scope of the project. Clearly if a
                                                                                      -’project is small and requires little in the way of resources, it may not be
 -’i                                     “. ,’
                                                                  ,? .’
                                                                                     : necessary to’conduct an extensive needs assessment. A general descrip-
-.1                                     ,u_
                                                                                     ’ ‘tion oft the most readily,. available, information on the apparent need
          .,                                                                        ,I -.could be q.uitesufficient             For example, a project designed to renovate a
-1                                  ,.           ,, ‘.r :                          .j small: apartment building in a neighborhood where the occupancy rate is
                             :                       ,.‘I           .:‘,,,t . ..* high, or’ where tenants were displaced when units were removed from
                                                           _. I .’ 1,     j             the inventory, could be justified without extensive investigation of the
                                                                                        overall level of need or the advisability of investing in other project
                   :,            i “’ ,,!,).‘,,I_,. ,/ ,‘j , ‘,; sites: (However; a,major investment in a large-scale project designed to
            i’ ,I              ’ ‘, b                                 ‘.          : ; replaCeseveral thousand:housing units with newer units would require
         ., ’                                                    .,,           1, a,far!more, extensive,investigation                    of the need for that type of housing
                                                                         .: ‘: ; :in that’lodation,’ relative: to .other competing uses for the resources.
                              ” ,,_                    , ! .,, ,. j ,, : : ;,-                         ,,,       ,,         ‘: , ,.,
             a;:‘~, ,. ! ,:;< ;‘, ?.r,, ,! I; i$ ;:’ I                                            :‘cii’r, ,,:;          .: __/ j ,i, /
       st&ijs            ‘ifi r$&.g                     ,;‘,;I’;:. ,,‘, 1’.I;.‘l.(‘I:&mating,               the need’.for a Project involves two steps: (1) the definition
                                                                                 : :: of ‘a st’andardof what ought to be and (2) the measurement of existing
       &&&&g,;:’                                 ;I:‘, ,,,:i’ (“&‘{>~‘;’,;: : ,:_,
                                                                                  ‘1; eond~itionsri~Need:is thenthe,difference                between the standard and
                                                                                        existing conditions. The definitionof adequate levels of housing or com-
                                                                                        munity development can be defined by legislation or program regula-
                        ‘I ij,‘-c,’ .,,i“,“,i I: :‘l;;.,“,.:a~j ‘,I tionsj expert .ol%nion, the expect@ions of client groups or target
                     !         ‘,’ ,; I, // :(,,,a :,,,”                  .*‘Y‘, i 1,p,oyiulations,: or comparison to: the; level of housing or community devel-
               ;,                  :,             ,.I ,!i ~‘,1;;                   ,.17.,opmentavaillable to other. groups.
                    /’                       i ’           ,’          (I            I,         : i.i’,>i: ( iv..
                                                                                                               ,/,    8. ~, :            I
                            I                                                          S,t,anda.rds of what ought ,to be may be established through legislation or
                                   ,‘z              1,’                                .program regulations: For -example, 'HUD defines rental costs exceeding 30
                                             ,, .. Y’                        :       ! Dercent, of. household income as an ‘excessive rent burden for low- and
                .I             ,~ :,                 ,;                     2’ ‘“,moderate-income’ families; .Ifc standards have not been established by leg-
                     ,:’            ,)I’.,.        I_,               ;::,.;,,        , islationor regulation, then~expertopinion                 is a potential source for nor-
                      ,,,.            .!;‘:, ,/..               .; *,I’               : matiae Mzindards.c However; reaching consensus among experts on
                                                                                                              I ,, :L..’ / .i”       , ‘-)

              ;,;e’,,                            I:,‘,,,:
                                                                    Page 17                                          GAO/PEMD-90-9       Partnership   Projects Framework
                   -.___         --.-.. - .-_ .-

                                               Appendix III
                                               Criteria of Need

                                               acceptable standards for housing and community development (using
                                               such methods as the Delphi panel) may be expensive and problematic.

                                               The expectations of client groups or target populations as a standard
                                               against which to assessneed has the advantage of relevance to local
                                               conditions. These expectations can be measured directly by local
                                               surveys, focus groups and other structured group interviews, and key
                                               informants. Indirectmeasures include the use of services that are
                                               already available. For example, long waiting lists for subsidized housing
                                               may indicate a need for more low-income housing.

                                               While surveys and other, direct measures allow direct feedback from tar-
                                               get-populations about specific issues, they have some disadvantages.
                                               They are potentially complex and expensive. In addition, surveys and
                                               structured groups tend to be reactive Athat is, they arouse expectations
--i          1)                                among respondents that action on their needs will be’taken. In contrast,
                                               indirect measures may be less expensive and less reactive because they
                                               are based on existing information-. However, the disadvantage of indi-
                                               rect methods is that they were not designed to measure the criteria or
                                               issue in question and may have validity problems.
                     ‘.                  ,’                                         .
      ,,I                                     To define a standard of housing.or community development through
      , (.    r                               comparison,’ data must be gathered for more than one area or group. For
                                              example, the quality of,housing. in one neighborhood could be compared
                                              to that in surrounding neighborhoods or nearby communities. The use of
                                              a comparative standard of need,can be more costly than the alterna-
                                              tives, depending on the source of information. In addition, unless,,rele-
                                              vant differences between, areas or’groups are specified and me:asured!
                                            I this’ approach;can neglect unique characteristics that invalidate the ,com-
                                              parison. For example, the housing needs of two neighboring areas may
                                              differ.                          I,
                                                  .. /                ‘,        ,.’
                                              In,general, the standard to, be applied depends on the program and the
                                              intended use of the evaluation Legislative or regulatory standards are
                                              likely to be preferred for their obvious utility in linking project objec-
                                              tives to program requirements. However, if an evaluation is designed to
                                             test the equity of program delivery, it might be more sensible to com-
                                             pare the need in the target community to other communities, disregard-
                                             ing the existence of program definitions of need. In this case, the
                                              additional expense involved in ascertgining the levels of comparative
                                             need could be justified. In.any case, the development of standards of
                                             need can be iterative with changes or refinements occurring as data are
                                             collected and analyzed.

              ‘.           .‘,              -.Page18                             GAO/PEMD-90-9   Partnership   Projects   Framework
                                                                                              Appendix III
                                                                                              Criteria of Need

                                                  ,’                     ./,

                                                                                 ‘Defining a standard using one’of the approaches above is only the first
                                                                                  step in assessing need. Existing conditions and services also need to be
                                                                        _)        measured and evaluated through comparison to the standard. The
                                            .’                                    assessment of existing conditions and services is the focus of the follow-
                                                                                  ing ‘discussion of indicators and measures for the criteria of need. We
                                                                                  identified criteria that are relevant to evaluating the need for the part-
                                                                                  nership project. They are problem magnitude and duplication and
                                                                .-        ,..
                                                                                  appropriateness. Magnitude refers to the size and distribution of the
                                       .’                              ‘.       ’ need.   Duplication*is concerned with whether other public or private
                .1      ”                                                         resources are.sufficient to address the problem adequately. Appropri-
                     /1 ~,,,,’              “.                                ,, ateness is whether the partnership approach is the most effective
                                                                                  method for meeting the need.        ’
            :               ,;              .,I                 \                  I-,                           ,,
                                                                                          In table III. 1, we Present some indicators and measures of the size of
        Srobiem Ma$nitud&.,                                                               housing ,and community development needs. Data on some of the mea-
                                                                                          sures are collected by the Bureau of the Census, the Bureau of Labor
                     : ‘I             .,,              I
                          ‘: ,.                                                           Statistics,       a&the Delk-tment of Labor’s Employment and Training%
              ”          :‘, J,’ Ii                           ..,                         Administration. ,However, the data may be outdated, aggregated to irrel-
        :i                                :                 ;: I,           /
                                                                                          evant geographic areas, not accurate for small cities, or not available for
               IS. ,,/ L                I,’                         ‘_’ ,’                geograp.hic, units smaller than a city. This issue needs to be examined
                        ; 4’                  i I”, ;                   :...:          :,and     the,data          supplementedi’if necessary, by original data or data from
                                                                     ,.,,             : ,‘, ,_,,    ;
                   .,                            ,. ,’                                   ,alternative’ sources; depending on’ the problem. The measures given in
                                                                  9,              -,
                                              ,’:                                       the table aepear as ‘magnitudes; but these should be compared to the
                             \,                         ,,.,; ! .‘,
                                        . ‘J ‘I                                        ” standards defined as we discussed above. Again, data requirements may
                      “.                                          i .J1 I(
                                                    , I,,                               not be extensive if the project is small in scale or only involves one or
        ,,         ‘: ,I         ,mj: I ;                                        ”                                                   ”       ‘,’ .. :

                                                                       .\!f I                  /,       ,( .            ‘,       ”
        8,,.,, /,,.;T                             .                                    ,,                  ..I                                ‘.    I.,
                                  ,, s                                                                                ‘.                                ‘.
           _’ .r:,, ; I:,’                  ,; (\ J           ‘.; ,’ i                                                      ‘,,        1          ,‘.
               j ,: ,,                     ) ”,!> /;               ‘.                                          ‘,,
                      ’ ! ‘;/             ., ‘, : :             ,:_‘s’;.                 ‘.          ,. ,,,
           /ii .‘>‘I,‘,, j., ,I ,,                                                                                                        :.
                                                                ‘+’ ,                        5,’                            :
                                  .i                                          .,           ,                               ..                1
                                                                                                                                   .             .’
                                     .,I           ,’                                            /,
                                                                                   \)             :                                                   ,..
                                                                :         .;                         ,’      .’                        : .,) ‘,
                                      >,                                                     :, .’                                                 .,
                                     .’                     ’                                                                            1,’
                                                                      .’                                                                          /
              !‘,’          ’ ‘i ( c ,, I,                                 :                               ,,
          ; ,:a               ,, ”                                                    I_.
                                                                                                                    ”         ,.‘.

        ,                          .                                                                                  : :
                     _’    i;;:,                  ,i,‘,    J    ,,   >         ’
                                                                                          Page 19                           GAO/PEMD-SO-9   Partnership   Projects   Framework
                                                          Appendix III
                                                          Criteria of Need

Table 111.1:Need Criteria: Problem Magnitude
Criterion                         Indicator                                              Measure
Magnitude   of housing            needs   Extent to which housing           is not       Ratio of existing stock to number of households;       number of new
                                          available                                      housing permits issued and starts and completions;        units lost to
                                                                                         abandonment,     fire, or demolition; rates of household formation
                                          Extent to which housing           is not        Proportion of household income going to rent; rates of
                                          affordable                                      homeownership;      interest rates for home mortgages;          ratio of shelter
                                                                                          beds to homeless population
                                          Extent to which      housing      is of poor    Extent of housing with inadequate        plumbing, inadequate         sewage
                                          quality                                         disposal, incomplete kitchen facilities, structural problems (e.g.,
                                                                                          leaking roof, holes in floors or walls), common-area        problems (e.g.,
                                                                                         .broken or missing stairs, no working light fixtures), inadequate
                                                                                          heating, lack of electricity or electrical deficiencies,     fire hazards,
                                                                                          inadequate   light and air, or signs of vermin; age of housing; extent of
                                                                                         overcrowded      housing; quality of management         of rental units;
                                                                                         condition of neighborhood        (abandoned    structures,   littered or noisy
                                                                                         streets, drug-dealing,     street crime, other physical and social
                      .       .                                                          conditions)
                                          Distribution   of housing      needs           Concentration    ofIhousing need’by      geographic     area or by
                                                                                         demographic     characteristics
Magnitude of.community                    Extent of economic       distress              Percent of people at or below the poverty level; per-capita or
deve!opment  ‘needs                                                                      household income; rate of growth in retail and manufacturing
                                                                                         employment;    unemployment     rates, rate of long-term unemployment,
                                                                                         or underemployment     rates; new capital expenditures    (investment     in
                                                                                         new plant and equipment);     amount of retail sales, service receipts
                                                                                         (income from the service sector), or wholesale trade; number and
                          I                                                              type of businesses;, crime rates by crime type, drug-dealing,      street
                                                                                         crime, and other social conditions
                                          Extent of physical     distress                Extent and concentration      of condemned   or abandoned  buildings;
                                                                                         extent of garbage-littered    streets; number and extent of unpaved              or
                                                                                         broken streets or cracked or broken sidewalks; percentage        of
                                                                                         streetlights missing .or ineffective; extent of inadequate drainage
                                                                                         and sewage facilities
                                          Distribution   of community                    Concentration   of community     development      need by geographic       area
                                          development      needs                         or by demographic    characteristics

                                                         In the absence of timely census data at the geographic level of interest,
                                                         information on problem magnitude may be available from the annual
                                                         household directories maintained by a number of private firms. For
                                                         example, the R. L. Polk Company provides urban statistical data as an
                                                         adjunct to its annual household and business directories in many major
                                                         cities. The data collected by the Polk Company have the advantage of
                                                         being available for household units, not aggregated into census blocks,
                                                         block groups, or tracts. Many cities, such as Memphis and Boston, have
                                                         developed neighborhood management information systems. However, if
                                                         neighborhood geographic boundaries have shifted over time, the data
                                                         may be aggregated to an inappropriate geographic area. National demo-
                                                         graphic updating services such as National Planning Data Corporation
                                                         provide current population and income data by census tract and zip code

   .,           .:’                                      Page 20                                               GAO/PEMDBOd         Partnership    Projects    Framework
                                                                            Appendix III
                                                                            Criteria of Need

                   ,,               6.
                                                                          primarily to large newspapers, ,banks, and insurance companies. :These
                                                                          services rely heavily on feedback from local planning agencies and local
                                                                          statistics. Other possible information sources are city planning commis-
                                                                          sions and other local government records, annual citizen surveys, and
                                                                      .., neighborhood advisory boards.

Magnitude of Housing                                                         We identified three major indicators of the magnitude of housing needs:
Needs                                                                        availability, affordability, and quality. These indicators are interrelated.
                                                                             For example, availability is the interaction of demand for and supply of
                                         .,                                  housing. But demand for housing is influenced not only by rates of
                                                                             household formation and population growth but also by affordability in
                                                                             terms of housing prices and household income. Similarly, housing supply
                                                                            is ‘a function of both additions to and reductions in available housing.
                                                                             Losses in housing may occur through abandonment, fire, or demolition,
                                                                             ,which are related to housing quality.’
                                                                 The-distribution of housing problems is a fourth indicator of the magni-
      ,..   1) J,                                               ,I,
                                                                *tude of the;housing need; Distribution refers to the geographic location
                                                             : ,_
                                                               ‘of the, problem and, the demographic characteristics of the population
             (!                    :                      ‘,. 1experiencing the. housing need. Distribution is also related to the other
   ~,                ‘_                                     I ’ indicators. The concentration of. need can exacerbate other problems
                                                            : through ‘!neighborhood-effects.”                For example, a deteriorated housing
          /                                   :                 unit. reduces the value of not only ,that unit but also surrounding units.
                                                                Thus, if maintenance is sufficiently costly, there is no incentive for indi-
                                      /I,,                      viduals to maintain their property. Any improvement in the value of the
  ,1               ;       . 1, .“_ ;                        : individual’sunit,wou!d           .be overwhelmed by the surrounding,
         j, <,; ,,,,, ‘,” i; i; ::- ..,,)j ,,,.‘!.‘. : ;\‘. ,;‘un~ermaintainedl.prope~ies.:               :I. c
                        :,                   .,         I”                   ,\j$,/ ,;I_’ a ( [ ; ,,:, *I.i
Extent td.~WhichHous&                        Is Not ’ : / Housing availability can be measured as the ratio of existing housing
Available                                                      ‘units to the currentnumber             of,,households. However, in order to inter-
                                                                pret current housing availability, information on changes in the availa-
     .,,        .,          :‘;                    j,’ <’, , bility of’housing and the number of households is also ‘needed. Changes’
                         i ;_<                    1 !           in available housing can be .measured indirectly by the number of new
             ,’                 II                              housing permits issued, housing starts, and completions. Of these mea-
                                     :‘.                        sures, housing. completions is the most valid measure of actual change,
          I.                            :                       since permits may be issued without subsequent construction and starts
                                                               may occur without reaching completion. However, it may be easier to
                                                                 !                                  ,’
                                           ./ ..‘,,         i ‘Some might include al 1 depreciation (both ph@cal and monetary) as a measure of loss. However,
                                                                            we have omitted monetary depreciation   here because we are concerned with actual physical loss, as
            ..’~                                    !I.                                                              ‘s:
                              ..1                              ,,           opposed to loss in value.

                        ‘:.    ;         ..,   “’                           Page 21                                        ,GAO/PEMD-90-9     Partnership   Projects   Framework
                                                Appendix III
                                                Criteria of Need

                                                collect data on housing permits, because data on housing completions
                                                may not be kept by local governments.

                                                None of these measures takes into consideration the loss of housing
                                                stock through fire, abandonment, and demolition. Thus, in order to
                                                determine net changes in housing supply, the loss of housing stock
                                                would also have to be measured. Local tax records or demolition permits
                                                may be sources of information on demolished or abandoned housing.
                                                Fire loss data may be available from the records of local’ fire depart-
      .’ :                                      ments and insurance companies. Local utilities or water departments
        ‘:                                      normally keep’updated records of water, gas, and electricity cutoffs,
        , ,.‘.;                                 which would permit an’up-to-date count of housing vacancies.
          8. _I,                               I:?$general,ihformation on housing availability may be accessible from
             ,‘, .,          ‘,                 ‘local government records, such as building permits and property tax
                                                  records, or the decennial census. Although census data are available for
                                                  decennial years and are very comprehensive, they are soon outdated
     .,                      :,                  ,and thus ,of limited utility for local planning. For this reason, the use of
                                                 local government records may be more appropriate. If local government
                                                 records are not-available or notvalid, then more expensive methods of
                                                  determining housing availability ‘may have to be used. For example, an
         /:.                          1 / ,’      evaluationsof the-Local Initiatives Support Corporation used key infor-
     ‘. ,         %                              mants in‘a neighborhood to assesschanges in the availability of housing
                       ,..                       by indicating changes on maps. (Vidal, Howitt, and Foster, 1986)

                                                  The vacancy rate, while available from census data, is not included in
                                                  the list of suggested measures for housing availability because vacancy
                            ,:                    rates appearto vary donsiderably, both cyclically and across locations.
                                                  In addition, vacancy rates seem to reflect imperfections (such as the
        ‘. ’ .,I                      ,, /1,, ,’ ‘time and;dost of seardhing for housing) in a housing market rather than
    ,.;                                   z housing. availability. (Pozdena, 1988)
              ., I’: ‘,,‘:’      ,             ‘,!/’ ,* : ,: <..,?,.1”
                                                                     :. -,     ,/,r
Extent to Which Housing,tIs Not                  ‘Measures of the: affordability of housing are different, depending on
Affo&,ble           ,        : .              * whether the lfocus,is rental’or owner-occupied housing. An affordability
                                            : problem in rental,housing can be measured by the proportion of house-
                                                  hold income spent on rent. The,magnitude of the need can then be deter-
                               ;   ‘,             mined by comparison to HUD'S standard for excessive rent burden for
                                                  low and moderate income households: rental costs exceeding 30 percent
                                                  of income. The higher the percentage of low- or moderate-income house-
                                                  holds paying more than 30 percent of their income for rent, the greater
                                                  the degree of need. Information on rental costs is collected in the decen-
                                                  nial densus. However; income data are only collected from a sample of
                                                  the population. Other potential sources of information on rent burden

,:                                                                                   GAO/PEMD-90-9   Partnership   Projects   Framework
                  .’                           .Page 22

                                                     Appendix III
                                                     Criteria of Need

                                                     are the records of local housing regulatory bodies or household data
                                                    .compiled by other surveys (such as the R. L. Polk data discussed above).

                                                                       An affordability problem in owner-occupied housing involves a number
                                    ;                              _,.of.factors, including the availability of mortgage loans from local lend-
                                                                       ers, downpayment size for low and moderate income homebuyers, and
                                                   I.                  the “affordability ratio” for homebuyers. Data on the availability of
                                                                  ,. mortgage loans by race, sex, income, and census tract soon will be acces-
                                                                       sible through the provisions of the Federal Home Mortgage Disclosure
                                                                       Act of 1976 (Public Law 94-200), ~~l~~~~~~t~~e~~~~~~~~to
                                                                       &nXSi!@Zg~lenders in an attempt to document the mortgage availabil-
                                 :, ‘.             ‘,’                .ity of, specific targeted populations. Low down payments for low- and
                                                            i, ,, moderate-income homebuyers are often obtained through mortgages
                     ;,                                           ,’ supported by the Federal Housing Administration and the Federal
-1                                                            ,! ,National Mortgage Association. (These programs are not described here,
-4                 :             ;              ‘,             ,, I but data on them could be useful in measuring need.)
                     :                  ‘. (.                   v’,,
          :                                 ,“,             :          The,generally, accepted ‘taffordability ratio” for homebuyers is no more
            ..,,.                        ,.,:,           ‘I,.          than 28,percent. of gross income or 35 percent of total installment debt
                                                           ,’          applied to mortgage loan,payments, real estate taxes, and homeowner
         ,                                                   ‘.    ,_  insurance.  The.evaluator would need to know current housing prices in
                            ;,: ,A, ’               ,. ._,,_ anarea, and’would have to compare them to median family income in
                                                                       that area to const,ruct this ratio. Information on homeownership is avail-
                                                                       able in the decennial census and the R. L. Polk data. Local property tax
                                                                       assessmentrecords are another potential source of data on
                        .’       ..(,                     ;           ‘homeownership.                ,,
                                    ‘/ ;                             i /,/.\,’         I,     ‘.
        ,. ”             ‘,       ‘/_. ._ : ;: ,,.                     T.he:demarldifor ,emergency;and transitional shelter beds can also be
                       ‘I          ‘.j/           I ,# /‘_‘I(,,!,,.hthought ofas a,measure oBhousing affordability, because a high
               ‘, ” J,‘,./ ;..,- :,_ ‘I ,I ,!.’ ‘,, *demand for such.seivices would,suggest a shortage of affordable perma-
                                                                       nent housing for rent or purchase. Specifically, shelter records could be
                                                                    r reviewed to, determine the percentage of available space used,and ,the
                                              s> ’                    ,average number of people turned aWay when sh,elters are full. Depend-,
                                  :           ,  >     .A(            ing. on the quality of. records kept by the shelters, analysis of the use of
                              ,’ .:                ,.’ !              the service can be inexpensive and quick. However, this measure has the
                      ,’                                             ..disadvantage of being linked to a specific solution rather than to a prob-
                  ‘. ,’                                                1em;for which several solutions may be considered.
              ‘/                         :,                             I, :
     Extent to Which.Housing Is,of                                    .Housing quality has two dimensions-the housing units themselves and
     Poor Quality~                      ‘. ‘.                   ,. ’ the condition of their neighborhoods. Most of the measures listed in
                 :,              .,,         “I                     ctable III. 1 are:drawn .from HUD'S definition of physically inadequate
                     ,’ 8..                                            housing. They are based on measures included in the American Housing
                                               .,’ (’                 Survey (AHS~ formerly the Annual Housing Survey) conducted by the

                          .‘I   I          -‘,      Page 23                                     GAO/PEMD-90-9    Partnership   Projects   Framework
 i                                                                                                                                                      ,
                                                      Appendix III
                                                      Criteria of Need

                                                                       Bureau of the Census. While the sample used for AHS is too small for
                                                                        estimates of housing conditions at the local level, the decennial census
                                                                       includes questions on local estimates of housing conditions, overcrowd-
                                                                        ing, and the extent of plumbing facilities that can be used. If additional
                  ’                           ,’                       measures of housing quality are appropriate, local communities could
-i                                                                     use the AHS questions to collect their own data. While this would be more
                                                                       expensive than using census data, a locally administered survey could
-I                                                                     provide more complete and current information. Other sources of infor-
                                                                       mation on housing quality are records of building and zoning code
                                                                       inspections and housing code violations.
                                                                       The quality of housing management was indicated by our expert panel
                                                                       as an important component of the,overall quality of rental units. One
                  ,I ‘,              ,’                               ‘reason,for the importance of management is management’s responsibil-
                    ,./ ‘,                                             ity .for maintaining .physical quality. Proxy measures of the efficacy of
                                                                       management include the timing and effectiveness of management
                                                                       response to tenant reports of maintenance problems. This information
                .’              ‘>.                                 I My be available >fromthe administrative records of the managers of the
               .,:-:- ’                               !.              ‘structure; Tenant surveys are another possible method for gathering
                            /‘,        ‘5 .“ ,,                        this information but would8probably be more expensive and time-con-
                        ; ‘.’ ‘-j1 /                        ./         suming than record,reviews,:in,addition to having the potential side-
               :~i”           ‘I” ,,!                                i effectof raising tenant’s expectations for changes that may not be
              ” ,L                                 ‘,                  fo&-gbmin&              !    !          :
         ,I         !i,            ,,.                                      ’                             ’ ‘.
                                 ,\ j                             ,’ The’second d.imension of housing quality is the condition of the neigh-
                                                                       borhood, which affectsthevalue, of the housing. This includes aban-
                                                                       doned buildings, littered or noisy streets, drug-dealing, street crime, and
                     ,,*,+, ”                             ;;.. ., other physical! and socialconditions. The quality of the neighborhood is
                      C,i(‘, ‘ ,‘,;, .,I, ‘i .: ,-$I’; ,,,,: alsoa measure for community,,development needs and is discussed
      ‘,. ::1,* “!,,;.,i !I ,,;P,. ‘rr   .IC‘,,,,‘,t-‘,.I,i.., /“, .below interms,of the!magnitude.of,community development needs.’
        ),.” :b:: 1, ) / .I.,‘,: ,.: .; s;,,.:,         i.)I .,      .;.I ,.:_.j) ’ ‘,i : 1 !.!‘i : :t.“; ,I! ‘,..I
                                                                    ,,,                                          ,!
     Distribution of. Housing Needs:                                   The’ distribution of housing needs is important becauseit provides a
           ,;;,.);,” ‘j j’            ,. ;( :j ,! : : + baseline’!for :asseSsing‘a project’s successin reaching a target area or
        ‘r <q/” .;                      /,,i         .G!’ ,            population:‘In.determining,thetarget area or population of a project, it
         ‘/ ) .(                     ‘,.d, ’ ,:            ;‘, : 1is important to distinguish’problems related to a place from those
                                                                 : ‘related to:people. .Each approach, targeting by geography or targeting
                                                                 ’ by population;has its advantages,and disadvantages. For example, a
                                                                       geographic target for a project intended to assist people may have lim-
                                                                      -ited effects in terms of the population served. Geographic areas defined
                                                                       as,low-income based,on an average income level below a poverty thresh-
                              ‘I                    ‘, :i              old may include residents who do.not have the characteristics of the
          .;; ,( ,‘, , ,‘,.I                      ‘I ,’ ,. “_ population that a specific project is trying to address. Yet those
                                   ,2   :.’      ,..,  ,‘, ,t,:I       residents, b’y dint of their residence in the neighborhood served by the

                               /.                ‘,   ,Page 24                                /. :‘GAO/PEMD-90-9   Partnership   Projects   Framework
                                                                                  Appendix III
                                                                                  Criteria of Need

                                                                                  project, may also benefit. At the same time, poor residents in neighbor-
                                                                                  hoods with an income level above an eligibility criterion may not be
                                                                         Despite the disadvantages of geographic targeting, it also has benefits,
                                                                         such as administrative convenience and efficiency in addressing neigh-
                                                                         borhood effects, resulting from the concentration of need (discussed
                                                                         above). Measures of. the geographic distribution of the need include
                                                                         identification of the boundaries of the area experiencing housing
                                                                         problems, description of the area as rural or urban, and estimation of
                                                                         the population density of the area. Data on the geographic boundaries of
                                                                         an area experiencing distress :may be obtained through key informant
                                                                ,.,I,    surveys, or observation. In addition, where census data relevant to spe-
                                       i                                 cific’measuresare available at a block, block group, or tract level, such
                                                               )’        units can be comljared on differentmeasures     of distress to help deter-
                       :    ‘,                                        ‘I mine the boundaries of the areain distress.

                                                                            I The description of the;population most in need by demographic charac-
                        ,,                                                    teristics is useful fortwo’reasons:  First, in evaluating the outcomes of a
       ,i         B,,.-,.i,) II                  I.                        I project, a combarison ‘of the, beneficiaries to the population in need pro-
                                            :                          :      vides a.basis+for judging the efficacy of the project. Second, other
                                                                              ,pcoblems or needs compounding the housing need may be identified.
   I                                                                          Descriptive statistics on measures of the distribution of housing
                                                     ):                       problems in the population can be obtained from public records and
                        .              ::                           -.   .’   reports, suchas the decennialcensus of population and housing or
                                                                              household directories maintained by private firms.

Magriitude ‘of;C$$r$nit;y                                        ) A needfor a.community development project may be indicated by the
                                                                   extent :of ‘economic .distresa or ,physical distress experienced in an area
   ,i                        N$$+‘.’                  :“!‘:‘.‘j’ ~~~,,orby~group~,  i .,;: :,‘! : ,i t-I _, ,I
            ‘,’    1   ,’         .’            ,.        ,,        ,;,       ,,,;,,   ,’   I’       _‘,   4,   ,/,   !,,,fI   ;,!,“’

FAAnt of EconcZic Distress      ‘,                                              ; There ‘are several measures of economic distress. For example, HUD uses
             .. r       -“, c $ :                                               ,.poverty rate;.per-capitaincome;        rate of growth in retail and manufac-
                                                                                  turing employment, unemployment, and long-term unemployment to
                                   I’           ‘P”                               determine the.eligibility, of local projects for the
                                                  1,                      l       Action Grant .program. In a previous report, we d
                                                 ‘....I                                    -
                                                                                &i??% ‘an-2                 that althoughthere are weaknesses in each one,
                                                                               s such as sampling limitations and outdated data from the 1980 census,
                                                                                  they generally provide ‘valid measures of distress. (US. General
                                                               I                ..Accounting Office,, July 1989)          c
                                            ‘.’                                                   :, ,_;          ,’
                            ..’               ‘.                            .,      t ‘. ,           i .’: :. ?i

                                                                                ’ Page 25                                               GAO/PEMD-90-9   Partnership   Projects   Framework
                                                 Appendix III
                                                 Criteria of Need

                                                 In addition, we reviewed alternative measures of distresssuch as new
                                                 capital expenditures, retail sales, service receipts, wholesale trade, and
                                                 the number and type of businesses. For example, in the report we state
                                                 that a decline in retail sales can serve as a proxy measure for “urban
          .                                      blight, lack of economic opportunity, and detrimental living conditions.”
                                                 Declines in retail sales have been strongly linked to population decline
                                                 and reduced income levels. Another measure of economic distress-ser-
                                                 vice receipts-has been linked tothe economic characteristics of a resi-
                                          .(     dent population.
             1 .’   /
                                                 Both crime rates sandcrime types are also relevant indicators of the need
                i    /;                          for community.development. For example, drug trafficking and street
    0.. ‘.                                       crime have made many urban neighborhoods across the nation, ,unsafe,
        !                 $                     ‘lowering local housing values and’depressing economic.development.
      ,,!                      ,’                Increases in this kind of criminalactivity may indicate a need for not
                                                 only .action against it but also neighborhood revitalization projects. .y
      ,.’              ‘, I .. ‘.,                ‘AnnualSdata on crime rates are available from Department of Justice
    I:’ *           .’      “..’ ,.                Uniform Crime Reports .for the. United States. While readily available,
    ,’ ,,.‘                  ‘;.    ‘t            ,both thesesdata,and the raw data from local police departments must be
                                                   used ,with caution,. They suffer from well-known weaknesses such as
                 .‘,          :                 i undercounting sandthe lack of uniformity in the definition of particular
 i .,            L : “I”‘., :: ‘,.! :/,:‘,t
              .;/,/..,:                        ,’ ,crimes. However, there,is an alternative or complementary measure of
                                                   the extent of crime. The percefitions of local residents about crime in ”
            .,         8                           their neighborhood are relevant. and could be gathered through a local
                               .,                  survey.                          ;,
Extent of, P+jrsicakDi$r&s                               Measures of: physical distress include the extent of abandoned buildings,
      .: ,, 1( /! :I) ,,, .;..;;,y ‘, :I ,,, I I ! garbage+littered streets, cracked and broken,sidewalks;: unpaved or bro-
   :,‘/I :;i,‘,c,,’              i .:I’., _. I,. ” ken aoads,,misSing.:orineffective street lights, and inadequate~sewage
                  : I “,,
                                                         and drainage facilities, ‘among others. As described above, the deteriora-
                                                         tion of the physical infrastructure of an area may compound community
      Ii,)          ,‘i’, /Y                ‘“1., 0’ ‘, development ,and housing problems’by driving down the value of hous-
     L .,, ,a _,,     :..i<   :,        1: ,,/           ,ing units and ‘making commercial investment less attractive.
           ., :                                             (,:‘.                              /
      ar&..ra.>    _L_. ...TLi.    :..                   Data     on physical, distress are probably obtained most easily through the
                                                         observation of existing conditiqns. Observation has the advantage of
                                                         being direct rather th-an reported. For example, some cities have used
                                                        trained observersto rate street cleanliness. The expense of training and
                                          /              using such observers depends in part on the frequency of ratings and
                                                         the need for a complete enumeration instead of a sample. (Urban Insti-
                                                         tute, 1980) Another source of information on physical distress could be
                                                         local government records of citizen’s complaints.

 ., ‘,              ,, _.           ,.1         ‘,Page 26                              GAO/PEMD-SO-9   Partnership   Projects   Framework
                                                                      Appendix III
                                                                      Criteria of Need

  Distribution of Community                                           Assessing the distribution of community development needs is similar to
  Development Needs.                                                  assessingthe distribution of housing needs. It facilitates the identifica-
                                                                      tion of the target area or population for the partnership project and pro-
                                                                      ,videsa baseline against which to evaluate the successof the project in
                                                                      reaching its target areaor population. Measures of the distribution of
                                                                      need ,,arediscussed above in relation to the distribution of housing needs.
                                                                      Information on the distribution of need in an area or population can be
                                                                      obtained from census data, directories maintained by private firms, or
                                                                      community surveys. Key-informant surveys and observation methods
                                                                      (in the case of physical distress) could also be appropriate for assessing
                                    .’ ‘,                             the-geographic concentration of a need.
  ,,        i. i                         (I/
    DuplicAtioh ahd                                                While magnitude refers to the nature and distribution of the need to
..-..A$pypoL*ig~e*&~,;,                           ..,.1 I _..;-! which      _a proj~~~~~~~p~~~~g~.~~~~~~~~~..~d~pp~opr~a~~~~~s.-~e-co~~
                                                                    ..-..--..-  r..
                                                                   cerned with the,nature,of the ‘response.(Seetable 111.2.)Duplication
                                                                   underlies the.question of whethera partnership project duplicates or
                _’                                                 substitutes;for other lresources.,Appropriateness involves the relevance
                                                                   of ,the response to,the need that has been identified. Such measures are
                                                                   necessary to judge the efficiency of partnership projects as a vehicle for
                                                               I. housing and community development.
                               ,,                               ‘.),’       .’ ,‘: ,,       ., ‘..,
 Table 111.2:Need Criterik Dup!ication and
 Appropriateness                                                    ln$cat&                                                          Measure
                           ~                                        Extent of other programs and,projects to                         Extent of other projects and programs
                                                                   ‘addresq’,needs                                                   available to address the need; includes
                                                                   ,, .‘,                                                            accessibility, capacity, comprehensiveness,
                                                                ‘.    ,/                            :                                and continurty of other projects or programs
         :,,:               ,l.., _, _, .:         .:‘,,,. :    ‘,’ ,Ex?ent
                                                                    ,            fo.which.need:will .be addressed,wjth,              Extent of this project’s resources to address
        ./,                       ,,i     “!                       ,”fhrs       ,“, :. /I,,, :: ‘. ,,,:’ /
                                                                      ; j :,project,                              .:;.,e, ,,         the need; includes consistency between
                                                                                                                                     projected results and needs and
          j.I   “8         P ,,,I            ,I .”                     .,,.
                                                                         .,   I       I’!*;,;  :    I,,     .,  : ,m.‘,
                                                                                                                     / ,. j  :       effectiveness compared to other solutions
                 ,:‘, i ,    ::              ): ,:. /‘I                 /’ ‘,! ;I          I :/ I,,.,      L’.. ,,.::
  .‘1,:I                  ,.        ” ,‘_                  ./
             I. ,/ ,.:                ,,I.     ;’        ,, Duplication occurs’if either the public or private sector is offering simi-
           (.‘I ,,,/ _;                         j,, ‘I,, .;,,,,lar projects or programs to those’proposed by a partnership. Part of
                                                               assessingthe overall need for,? ‘partnership project is identifying the
                                                               accessibility; &pacity;comprehensiveness, and continuity of other pro-
                                                              grams and projects that are already in place with the same or related
                                                              purposes. Compilingthis sort of resource inventory usually requires a
                                                 /            survey of ‘service ‘providers. But the documents and records of related
                                                              projects, are other ,potential sources,of information. In some instances,
                                                              local planning agencies may haveXready done this work in order to
                                                        ,, ‘, compile a service
                                                                            ., ‘:
                                                                                  .directory for local
                                                                                                   I, citizens.
       ‘~                                         ‘L ,,              ,”       ‘,        ,!,

                      ‘.                                              Page 27                                               GAO/PEMD-90-9         Partnership   Projects   Framework

                                                 Appendix III
                                                 Criteria of Need

                                    ”            Alone, a resource inventory does not reveal need. It has to be combined
                                                 with measures of the extent of the problems. The comparison of the
                                                 type, location, and accessibility of services to the type, location, and
                                                 magnitude of problems forms the basis for judging whether a housing or
                                                 community development project is needed. Only if existing projects are
                 ‘_        a,                    inadequate or ineffective should additional projects be considered.

                                                                    If it is decided that a project is needed or is not duplicating other
                                                                   projects; the next question concerns appropriateness. Appropriateness
             ‘.                                                     includes the extent of resources of this project to meet the identified
                                                                   needs. It includes the consistency,between the projected results and the
                                                                   identified needs. It could also include projected effectiveness at meeting
                                                                   the needs compared to other solutions.,,For example,.there may be.a high
                                                ,’                 rate of homelessness,but more emergency shelter is not always the most
                                            _                      effective or appropriate solution to homelessness.If in fact it is deter-
                                      :                            mined that the project is the preferred approach to the ‘problem, then
                ,. “                                               there is the question of whether the project could proceed without a
        8’                                        :          : partnership. In order to’determinethe need for a partnership project,
      .                                                            the’ extent of available public and private resources should be measured.
      I. 4’              ,‘,                                   ” For example, if ‘private investment in the commercial development of an
                                                                   area would have occurred ,withoutpublic involvement, then public
    3,. :~-‘-~;‘,I .‘,,(                                           investment in a partnership is substituting.for the private investment
                                             : .)/‘<’ :          that zwould.have occurred anyway.
                                       I.         ;: ”             Three possible methods of,measuring the extent of substitution were
       ,,                                                          discussed in the literature. One e&&ration examined what would have
     ‘?’ ,,                                                        happened in local projects if a large nonprofit group had not provided
     .’                 ,e ,/._.
                              :, .,.; ,,.,     I . j j : assistance’by developing descriptions of alternative outcomes through
         ““...z .,_ : 3,, 1, ,,                      ‘.,I
                    ,,( I.: ,, “0          i ,‘.          ;; i‘I” interviews with staff members’& the project. (Vidal, Howitt, and Fos-
      (, jt(lt: ‘,:I‘:!Ij; ;,‘:.        .,,, ‘I, I .:j ,, ter, 1986) A second evaluation looked at what would have happened at
    1 .I./ *. ..,. .                        ..a,-_,/,,, ..,
                                                                  the local level if“iJrb~‘~e%$$%ient Action Grant funds had not been
     ,, ; .:. ” : !                                               prqy~,dediby:ha~i~g,real estate:experts review project records and assess
            I.         -i < :,’               : “1 .,., .“).       whether
                                                                         :i.. the project( yould have occurred without the federal support.
                                 >;..,                            (I+man,~ lS$3?)A third, method of measuring substitution compares the
                                 ,I,                     ., observed rate of return to a private investor from an investment in a
                                                              , local partnership project to the mar,ket rate of return earned on a similar
                                        ,>‘:.                     private investment.   ’ IJowever, this method fails to take account of the
                                    ./                            nonfinancial factors that may motivate the private sector to become
                                                                  actively involved in a partnership? such as a desire to create a favorable
                  1’                                              l!$rblic,  image (or “good will”). (Abt Associates, 1981)
                                                             ‘,                            :
                                                                  &h of these’methods h.aspotential validity problems. For example,
                                                                  while the alternative outcomes were developed by an outside evaluation

:     ,l,L,.‘.        ‘2    ,i,-.       ,‘.     Page28                                   ‘:, GAO/PEMD-908   Partnership   Projects   Framework
                                           Appendix III
                                           Criteria of Need

                                           team, they were subject to confirmation and revision by staff in the local
                                           projects who were still receiving support from the nonprofit organiza-
                                           tions. The review by experts may be less biased but is dependent on the
                                           accuracy and availability of project records. Use of the market rate of
                                           return assumes that without the incentives provided by the partnership,
                                           the investors would have made a typical investment choice among an
                                           array of alternatives. Additionally, expected investment returns are not
                                           always realized and, thus, this observed rate of return is not always
                                           valid. Despite these potential problems, these methods are a promising
                                           beginning to the difficult problem of assessingthe extent of substitution.


;’   ,,
                           : ~:<                              /
                        ” ,: ,:,I          ‘,.; ,;
              ,’ / I’               _’‘_      ,: /

                                           Page 29                           GAO/PEMD-99-9   Partnership   Projects   Pramework
Appendix IV
 Process Criteria

                ./,:’                       ‘.          ‘,           The evaluation of the process that a partnership uses to implement a

                        q ./                            ,,           housing or community development project is important. Process vari-
           ,.:           ”                            ,’ ,I          ables have been linked to the successof partnership projects because the
                                                                     process leads to immediate results .as well as long-term outcomes. Pro-
         “/ ,!’ I’,,‘.                                               cessmeasurement, involves documenting staff time, resources, and ser-
                  ., ,‘( ,,, .‘)                ,         I          vices delivered,as well, as measuring effort. In an empirical study of
                 i: ”                            .,:               ,neighborhood development organizations, Mayer found that among the
                                              ,;’           ,,;. . ,prime factors ,determining the level of successof partnership projects
                           .’           ; ’                          were a skilled executive director, a key staff person with broad experi-
         I II /.                                                     ence and background,‘and a-track record of accomplishments1 He found
                                                                    that such process variables as teamwork, staff skills, and board partici-
               ’           I , ‘, : ;                .,’ ,.        :pation played a greater role in successthan the organization’s budget,
               .’ ,                ,, ,.                             age,.or staff size.,
                        .I .‘,IhI’I.                    _~
                         ;.                      ,’                 Information about the’nature of the actual program being implemented
           ,’         ;                                            ,is as important as information on outcomes. Process evaluations can
                                       1’                       I permit-decisionmakers and information users to understand the dynam-
                           ‘.        .;                ~ ‘.         its of program operations and can reveal areas in which programs can
               ,.                                       /,’         be improved as well as highlight the strengths of a program or project.
               .I I’                                                Patton notes that “a serious.look at the actual substance of the program
                                                                    being evaluated can prevent some . . . obvious but oft repeated evalua-
                 ,,                      .:..                       tion failures,” (Patton, ,1986)

                                                          The measurement of process variables may prove to be particularly dif-
                                                          ficult. Rather than discard elusive concepts such as “quality of manage-
                :                                        ment,” attempts should be made to define’and study them, using case
    :. I’,                          ,.       .,,         studies, qualitative methodologies, or innovative techniques; Measure-
                    :,                         ,         merit issues such as data quality, data availability, and data selection
          ,:      ,,.            ,1’                     will be addressed as,we.procee.d;follow-on efforts will analyze design
                        ,_         .,                    issues   in greater detail, as the framework is applied to actual
  .. .,;I,‘,‘:-                           ‘!
                      ;; . ,:,: :, ’ ‘; .‘:        ; !, partnerships2                    I     ,,
                                                 ;    .,                                   1

                                                                                                               ‘Mayer noted that “internal characteristics are of special policy interest in terms of both program
         S,.’           ’ ...I     ..,.:.,: I ,: ,.,.                                                          success and capacity building.” He grouped these characteristics into seven areas, five of which are
                                                                                             ‘.                key staff, short-term planning, management, long-range planning, and board of directors. (Mayer,

                                                                                                               2A combination of methodologies may be employed in evaluating public-private partnership projects.
                                                                  ‘,                                           For the Mayer study (1984), statistical and case study approaches were used in tandem. Grant appli-
                                                                       ‘.,,                                    cations were reviewed in order to obtain information on intended project outputs, funds leveraged,
                                                                                        .’                     and timelines for completion of project milestones. Information on intended outcomes was obtained
                                                                                                               from quarterly and final reports sent to HUD. Interview guides were developed for discussing the
                             ,.‘, ‘.                - ..1.               .,         :             , ,1        .organization’s work with key actors, who were selected from eleven categories.

‘.. ,I      (,,C                       ,,    ‘,,   1’        ,:           .,::.          ::       ,s:   ‘,   : Page 30                                       ,‘, .GAO/PEMD-90-9   Partnership   Projects Framework
                                                                          Appendix IV
                                                                          Process Criteria

                                           . .                            Caution is required in looking at the process data that might be gath-
                                                                          ered. Many of the data are based on subjective assessmentsor judg-
                                                                          ments (such as the honesty and integrity of management or management
                                                                          abilities). Some of the data can be obtained only by case study or direct
                                                                          observation. Data may be difficult to obtain because of legal constraints
                                                                          or unwillingness to speak frankly on the part of project staff. In addi-
                                                                          tion, no centralized data base exists and the data bases that do exist
                                                                          tend to be partial, incomplete, and unreliable. Again, it should be noted
                                                                          that in any given evaluation; not,all of the criteria listed here will need
                                                                          to be addressed.
           ,. ,, ‘.    .,,                                                     ‘,
          I ‘)      .’                               _.                    For the purpose of our framework, we categorized process indicators
                                                                           according to four criteria: planning, structure of the partnership, man-
                                                                           agement of partnership operations, and acquisition and management of
     ., *                                                                  resources. Planning refers to the initiation and process of starting a
                       )_                                                  partnership project. The structure of the partnership refers to variables
                  _ ,’   ‘!                                         1      in the organization of the partnership itself, such as the number of par-
          ‘../. ,,. ;;                                                    ‘ticipants and their skills. Management of partnership operations is con-
          ” 3’ “.           ,,                                             cerned with factors such as leadership, accountability, and coordination,
              ,‘,                                                        ‘,both within the partnership and with other entities. Acquisition and
               ;        ;..                                                management of resources focuses on the financial and other resources
                                                    :.                     necessary for the implementation of a housing or community develop-
                                                                           ment project.
     ‘5: ;,                       ,,
                                            ,, 1,        ‘_        ;
                                                               Two major planning steps occur prior to the implementation of a part-
                         8,. .’ (’ ,‘/.,_.‘:s                  nership     project: initiation of the partnership and selection and design of
          :.,                                                  the housing or community development project. Sample indicators and
           .,,. ,*          ,‘,‘“,     ,, .,           (‘i :, ;mea.+ires for these steps in the planning process are shown in table IV. 1.
              1’ ./ ,..:, ;/ (, .., 8.  : :, I, , I,.
                                                     ;          ‘:,Z-./,/ ;’ I     ‘,>;’ g,., .t .,., ” ,,
Table IV.l: Process Criteria:.,Planning,
                                                                          Indicator                                   Measure
           ..:,              ,.                               ,.
                                                                         ‘-Initiation of partnership project          Emergence of partnership initiator: reason for
                                                 ” ‘,,                     :                                          forming the partnership; timing of
                  /.’                  !                                                 .,                           involvement of participants; degree to which
                  )<                                                                                                  participants share common agenda
          .’                                                              Match of project type and complexity to     Quality and extent of use of needs
                                                                         abilities of partnership and community needs assessment, feasibility studies, and market
                                                                                                                      analyses; existence of plan for leveraging
                                                                                                                      funds; accuracy of time, cost, and resource
     ,,                                                                  ,Quality of planning efforts                 Degree to which planned activities are linked
                                       :                                                                              to objectives; documentation of goals,
                                       ‘,                          5’~   .,‘,                                         objectives, and implementation plans;
                                                                                                                      simplicity, directness, and feasibility of
                                                                                                                      project design

                                            I        .,:
 :                      .,
                                                                         Page 31                                    GAO/PEMD-90-9   Partnership   Projects   Framework
                                                     --                       -.-...-_ _._ ..- . -

                                                 Appendix IV
                                                 Process Criteria

Initiation                                       To document the initiation or formation of a partnership, one would
                                                 have to describe the process by which the need for a partnership was
        ,                                        decided and-the,initiator or catalyst for the partnership emerged. These
                                                 two variables have obvious implications for later decisions about the
   i,                                            structure and,focus,of the,partnership. For example, if the partnership
                             :’                  is initiated ,in,responseto a crisis, the planning process may be truncated
                                                 and the partnership may,be short-term. Partnership projects can also be
                                                 initiated in response”to a public program or incentive rather than a
                    ,                            demonstrated .needin the community.

                                                           There are several ways in which partnerships may be initiated and a
            .: .,                                         number of possible initiat0rs.A company could seek to become more
                     ,’‘I ,                                involved,in a community’ where it does business, for philanthropic or
                ‘,              ,’ : .’                    other reasons; For example, ,General Motors initiated a public-private
                                       ,;,                 partnership with Dineh Cooperatives, Inc., a locally controlled commu-
        .‘.                             :             I nitytdevelopment corporation in Leupp, Arizona, to establish a Navajo-
     .,,                ., : : ,..                         owned tool-and-diesupplier plant;:The project was a good business
      .,a(I                 ‘/             ‘et        l :. move for General Motorsandassisted in changing its reputation regard-
  ‘.          ;,,’ ”          I ,, .. ..,(, ‘,_      :’ ;:ing .the support of.minority: business start-ups and expansions. (Robbins,
         ,‘: 7 ! ,,!S/ ,’ ,(, ,,                     ‘. : -,1988)Partnerships can also be initiated by a mayor or a city official
            ”               8. .‘,‘,,’
                            ,..   !                  _i : interestedFin revitalizing,a- city, as in the Charles Center in Baltimore. In
                          “,.$./i:,. ,.;’ “’ ,I, : i     “:another instance; community groups could work with an intermediary
                                                           development association to .foster .a local partnership.

                                                                   Related to the question of who initiates.the,;,p,a$nership is the,ti,ming of
                 .,: /.,,,,,- ,‘,, ,-            \., ,., the involvementof other participants. For example, a partnership init& ,
       ,)_ / ,;                      ‘;;’ ; ,, L . ated!by the private, for-profit sector with the local government kc&id
      ,‘;...-,,, ,,,’! : I, ,/ ..,   // .Y ,: + ingcludecommunity groups,ata-later point if they discovered that some
     <’    \, 6,; :, ..‘.i:‘!:,, ;. ., ‘, i’, ,‘1 k grantsW,ere:not:available vvitho,utneighborhood representation. The
                                                                   timing ,of involvement may relate to the ,qu,ality of. the coordination
 .,/ .~,
i.P  ” .>I:;).?.
             ‘/I, ‘+,b’z2;;.
                           ;,,y..“L .‘:‘;j,;,;,+;,;,*:;,:
                               ‘7’‘.c,l              5.*y.y:,’‘;&.:.among-‘:‘sector~.dis~,ussedunderstructure ofthe partnership: r:+’ , .:,‘.::,. .’
                                                     :;z.:(.__,                                     ,.‘(_,,F.,
 \/ .;.‘,.“. ,, ;:1,,,I(,‘(f..,, ‘.,t..I, t,, ‘tic, The degree to which the participants share a common agenda can also
                        :, ic,/ I’,; ,: :, f
       ., i           j “);.I:,           ,(        I _,::,,, be important to the facilitation of the planning process, but often a com-
              /I          1,,,.,_.                      /i,. mon agenda may not be reached. A common agenda does not mean that
                                                                   evew sector has the same motivation for participating but, rather, that
                                        ,,                         their  different ‘motivations!lead them to the same action. There may not
                 ,. !                                              be a single or even an internally consistent group of objectives for each
                                                                  partner, and publicly stated goals may not always accurately depict the
                  “/                          ,’         .., actual goals of allparticipants: Thus, data on which to judge the com-
              ‘,              ‘,                  ‘,. i
                           -,,(. .,I , ,; .:                      mon goals of public-private projects are not easily revealed or retrieva-
         >                                             :I ,’ ble. (Lipman, 1988)
         ,. I ,., . ,,. /; ,I                  j 8’
                                                             ,.          ,,.r,,       ,.,      ,I .: ,,

                    I> (,‘    ,,   ;   .I:,     , Page 32                               i : . GAO/PEMD-90-9     Partnership   Projects   Framework

,                                 Appendix JY
                                  Process Criteria

    Match of Project Type and     The selection and design of a project is the second part of the planning
                                  process. A needs assessment can provide information on the scope and
    Complexity to Abilities of    location of problems that the partnership should address. Feasibility
    Partnership and               studies and market analyses performed by project staff can inform deci-
    Community Needs               sions on what kind of project is appropriate. Feasibility studies can be
                                  helpful in identifying the potential the project has for success. Market
                                  analyses can provide data on how large a project the community can
                                  sustain. These analyses do not have to be extensive and can involve
                                  neighborhood residents and business people in an effort to assist in
                                  assessing current conditions, defining pressing needs, and identifying
                                  targets of opportunity.

                                  In addition, accurate time, cost, and resource estimates are helpful for
                                  the projects under consideration, so that the partners can assess the fea-
                                  sibility of leveraging the needed resources to complete projects. The
                                  extent to which these tools are used in selecting an appropriate project
                                  provide.indirect measures of the,match of project type and complexity
                                  to the abilities of the partnership and the needs of the community. Data
                                  on planning can be obtained from record and document reviews and on-
                                  site visits and interviews with participants. Time, cost, and resource
                                  estimates would appear to be easily obtainable and are particularly use-
                                  ful for linking planned activities with objectives and objectives with

    Quality of Planning           The plan for implementing the project may be written in simple and
    Efforts                       direct terms, with clearly stated time, cost, and resource-estimates.
                                  While these are generally accepted standards for planning, under some
                                  circumstances clarity could result in conflict among the partners if it
                                  exacerbates disagreements that are difficult to reconcile. In some situa-
                                  tions, it may be advisable to formrather   general objectives, with the
                                  understanding that they will be made more specific as experience
                                  accumulates on the project.

                                   Evaluators have explored in depth the difficulty of reconciling goals and
                                 ’ objectives and their shifting nature over time. (Lipman, 1988; Pressman
                                   and Wildavsky; 1973) However; economic development, revitalization,
                                   and neighborhood improvements are lengthy processes, involving many
                                   groups, who can easily lose sight of project goals and spend energy
                                   attempting to solve problems larger than those at hand. Furthermore, in
                                   some cases, if the goals are explicitly stated, that statement may help
                                   keep the partnership project on course. (National Institute for Advanced
                                   Studies, 1978)

                                  Page 33                            GAO/PEMD-90-9   Partnership   Projects   Framework
                                        Appendix IV
                                        Process Criteria


                                        Process criteria that involve planning and initiation can be documented
                                        by multiple methods such as site visits, interviews, and expanded
                                        responses to questionnaire data. Mayer, in his study of 99 neighborhood
                                        development organizations, found that written self-reports, telephone
                                        calls, and site visits all contributed to information gathering. He also
                                        found that interviewing a large, varied set of actors inside and outside
                                        the organization resulted in a relatively complete picture of project pro-
                                        gress and its causes. (Mayer, 1984, pp. 223-24)

                                        As shown in table IV.2, the structure of the partnership is indicated by
    Structure of the                    three variables: composition, representativeness, and skills of the part-
    Partnership                         ners. Describing the composition of the partnership in terms of the num-
                                        ber of participants, their affiliation, and the stability of membership
                                        provides a context for interpreting other information on the structure
                                        and management of the partnership. Information on the composition of
                                        the partnership can be obtained from document review.

    Table IV.2: Process Criteria: The
    Structure of a Partnership          Indicator                                        Measure
                                        Composition     and representativeness   of      Extent of representation         of different
                                        partnership                                      constituencies;      degree and nature of
                                                                                         involvement      of participants    from different
                                                                                         sectors; stability of membership
                                        Skills of participants                           Skill in acquiring financial and other
                                                                                         resources; technical skills and management
                                                                                         abilities of partners; prior experience of
                                                                                         partners with joint ventures; political
                                                                                         awareness; influence and financial ability of

                                        The representativeness of the partnership, or the equality of opportu-
                                        nity.for different groups to participate, is measured by the extent of
                                        representation of different constituencies, the degree and nature of their
                                        involvement in the partnership, and the stability of the membership
                                        over time. The involvement of different constituencies has been identi-
                                        fied as an important element of partnership structure for two reasons.
                                        First, representation from different sectors can give the partnership a
                                        broad base of legitimacy, which may facilitate project implementation.
                                        Second, the involvement of traditionally underrepresented groups can
                                        result in their increased self-reliance and self-determination.

                                        The question of self-determination was important to the model cities and
                                        antipoverty programs of the 1960’s. According to Secretary Kemp, cur-
                                        rent HUD policy encourages self-determination in resident management

                                        Page 34                                       GAO/PEMD-90-9     Partnership   Projects   Framework

                 Appendix IV
                 Process Criteria

                and spreading home-ownership opportunities in public housing. (Wash-
                ington Post, September 17, 1989) In earlier programs, self-determination
                weakened support from some project stakeholders, such as large-city
                mayors, who saw emerging community or minority leaders as competi-
                tors. In public-private partnership projects, similar problems could

                Finally, participants in a partnership bring a variety of skills to imple-
                mentation. The skills of the partners can have a great influence on a
                project’s success. For example, partners with the ability to identify
                outside sources of funding for a project can reduce the financial burden
                on the partners themselves. Those with considerable financial or politi-
                cal influence are also likely to be successful in this regard. Similarly, the
                technical expertise of partners in housing or community development
                projects can fill gaps in staff abilities, such as experience in bidding and
                contracting processes. Past experience with housing and community
                development projects has also been linked to project success in obtaining                                 -
                funds and cooperation from different sectors. Again, these data could be
                obtained by record reviews, site visits, interviews with key informants,
                and the direct observation of partnerships,

                Table IV.3 shows indicators and measures for evaluating the manage-
Management of   ment of partnership operations. The indicators include leadership, coor-
Partnkship      dination within the partnership, coordination with other entities, public
Operations      accountability, and project implementation.

                3An example of the positive effects of self-determination   is the Neighborhood Housing Services pro-
                gram, which provided a segment of the population with some organizational skills and support and
                made it a substantial partner in a long-term effort to reverse neighborhood decline. Some research
                has indicated that resident involvement in the program provided low- and moderate-income residents
                with access to and some control over services and resources they otherwise would not have had.
                Resident leaders reported that the program gave them a sense of hope, a great deal of pride in their
                program, and, independence from the Neighborhood Reinvestment Corporation, the city, and a range
                of other programs that they felt had failed to help them. Mayer (1984) also reports resident participa-
                tion as a vital outgrowth of the neighborhood development organizations he studied.

                Page 35                                          GAO/PEMD-90-B      Partnership   Projects Framework
                                        Appendix IV
                                        Process Criteria

Table IV.3: Process Criteria:
Management of Partnership Operations    Indicator                                             Measure
                                        Quality of leadership                                Leadership’s  prior experience,   technical and
                                                                                             management    skills, commitment      to project
                                                                                             goals, and consistency    and stability
                                       Degree of coordination      and participation         Regularity of meetings of partners; frequency
                                       within partnership                                    and clarity of communication;        .extent to which
                                                                                             partners are included in decrsron processes;
                                                                                             process by which financial resources are
                                                                                             controlled and managed;        presence and
                                                                                             success of mechanisms        for resolving
                                                                                             disputes; degree of cooperation          among
                                                                                             partners; degree of overt consensus on
                                                                                             project operations and objectives
                                       Degree    of coordination   with other entities,      Extent of good working relationships         with
                                                                                             other agencies and of public relations efforts
                                                                                             to gain support for project; continuity of
                                                                                             liaison with neighborhood      groups; nature and
                                                                                             degree of responsiveness        to’community
                                       Degree    of public accountability                    Quality of recordkeeping;      nature and extent
                                                                                             of quality control efforts; honesty and
                                                                                             integrity of management;      existence and
                                                                                             quality of plan for evaluation
                                       Project implementation                                Flexibility or responsiveness    to changes in
                                                                                             circumstances;     use of procedural,   legal, or
                                                                                             regulatory shortcuts; time effectiveness       or
                                                                                             adherence to deadlines; degree to which
                                                                                             implementation     matches plan

                                       While not all partnerships necessarily have a formal leader, experts in
                                       the area of public-private partnerships identified leadership,as an
                                       important aspect of a project’s success.In some cases, the leadership of
                                       a partnership may consist of the extended ongoing efforts of a key indi-
                                       vidual who is not a formal leader. One of the primary roles of a leader is
                                       that of facilitator, bringing together resources, serving as a liaison
                                       among participants, and soliciting outside support for the project. The                                       I/
                                       specific technical or management skills of a leader can include the abil-
                                       ity to plan a project and the ability to assemble technical expertise, stim-
                                       ulate action by boards, staff, and funding sources and effectively raise
                                       funds. (Mayer, 1984, p. 101)

                                       Measuring the quality of leadership is difficult but can be accomplished
                                       through case study methods including site visits, interviews, and infor-
                                       mal questioning of other partners and participants. (Mayer, 1984, p. 99)
                                       Site visits could be timed to include direct observation of board and staff
                                       meetings in order to assessstaff ,management functioning. However,
                                       Mayer found that the worth of specific talents was best demonstrated
                                       by observation methods when directors were individually present or

                                       Page 36                                            GAO/PEMD-90-9     Partnership   Projects   Framework


     n                                    Appendix IV
                                          Process Criteria

                                          absent. That is,‘often the’only time Mayer could really assess how effec-
                                          tive were certain key leaders was when organizational leadership was
                                          taken over by a more appropriately skilled person who improved staff
                                          relations, made decisions, or provided new momentum to a project.

                                          Within the partnership, the work of participants from different sectors
                                          needs to be coordinated so that the benefits of participation by multiple
                                          sectors. are achieved and the potential for conflict is reduced, Measures
                                          of the degree of coordination can include regularity of meetings, fre-
                                          quency and clarity of communication, sharing of information and
                                          resources among participants, and the presence of mechanisms for
                                          resolving disputes. The frequency of meetings could be measured
                         :                through record data, written and oral communications could be sampled
                                          and evaluated by rating procedures, and a variety of unobtrusive mea-
                                          sures could be used to measure information-sharing      and conflict resolu-
                                          tion4 Sociometric measurement techniques could assist in measuring the
                                          degree of cooperation among participants, the degree of consensus on
                                          project operations and objectives, or the presence or absence of dis-
                                          agreements.6 This could -becostly if extensive observation over a long
                                          period is required.
             .I,.   .’                    Mayer noted in his study that

                                          “what contributed most to success was a board that worked eagerly and harmoni-
                                          ously with staff on shared objectives and included some staff with specific skills
                                          and contacts. Disagreements. . . dramatically reduced the potential for these kinds
                                          of assistance +. , and caused-significant,drains on overall organizational energy.”
                                          (Mayer, 1984, p. 114)

                                   ,:   ‘/. Mayer
                                              ;” : i cited specific examples of the
                                                                                    effects of cooperation and
                                           ” I,:,,:‘.‘,:    1,           :
                                          The’bartnership typically needs to coordinate with organizations and
                                          groug not ‘recresented in it: Coordination with agencies implementing

                                          4Unobt&ivkj   or nonreactive, measures are those “that do not require the cooperation of a respon-
                                          dent andthat do not themselves contaminate the response.” (Webb, 1960, p. 2) In this instance, such
                                          measures might be based on a review of minutes of meetings or correspondence among participants,
                                          as opposed to a survey or interviews in which the responses may be affected by the fact that partici-
                                          pants know their statements will be used aspart of an evaluation.

                                          %ociometric scales have been developed that allow for the quantitative description of group interac-
                                          tions. Miller describes Hemphill’s Index of Group Dimensions, Bales’ Interactional Process Analysis,
                                          Seashore’s Group Cohesiveness Index, the Sociometry Scales of Sociometric Choice and Sociometric
                                          Preference; and Bogardus’ Social Distance Scale; such scales are directly relevant to the issues we
                                          address. (Miller, 1970, pp. 200-24. See ‘also Mitchell, 1969, pp. I-50, and Whitten and Wolfe, 1974, pp.

                                          Page 37                                          GAO/PEMD-90-9      Partnership   Projects   Framework
            Appendix IV
            Process Criteria

             similar projects is important in order to avoid duplication of services to
             the same area or population and to take advantage of opportunities to
             make use of complementary resources. The partnership needs to coordi-
             nate its efforts within and between community groups. Community sup-
             port for local projects has been linked to successin raising funds. But
             community members may not support a project because they disagree
             either with specific aspects of the project itself or with partnership rela-
             tions between the private and public sectors in general. Thus, the extent
             of public relations efforts on behalf of the partnership project is one
             measure of the extent of the coordination of the partnership with the
             community. Other measures of coordination with the community include
             the continuity of partnership relations with neighborhood groups and
             the degree of partnership responsiveness to community interests. Pro-
            ject records, interviews, and direct observation should provide data on
            these measures.

             Public accountability is an important issue in the management of public-
             private partnerships, because by definition public resources are
             involved.‘Ho’wever, the accountability of the public sector may become
             blurred when it works with the private sector. The extent to which pub-
             lic accountability is maintained may be measured by the quality of
             recordkeeping, the nature and extent of quality control efforts, and the
             overall honesty and integrity of management. Except for honesty and
             integrity .of management, information on these variables generally
             should be available from project administrative records. The honesty
  .’         and integrity of ‘management may be measured by the number of formal
             complaints filed, evidence ‘of,federal or state investigations or legal
             actions, or the questioning of other key actors outside the partnership.
i,,                         ‘,
            ‘While public accoun&b$ty ‘is a concern in public-private partnerships,
             the blurring of public and pri&te sector roles may give the partnership
             more flexibility in implementing .projects than the public sector would
             alone. Flexibility in‘project design is also important. Project plans can be
             seen & tools for focusing initial work efforts, which can be updated as
             new information and expertise are gained. This kind of information can
             be obtained from records and’ direct observation.

            One key indicator of project implementation is flexibility, or responsive-
            ness to change by the partnership. An example of flexibility in manag-
            ing partnerships is the Wemgart Center in Los Angeles. The project
            stemmed from an original committee of 60 who wished to expand detox-
       .,   ification facilities’in the city. Because many of the homeless have social

            Page 38                             GAO/PEMD-90-9   Partnership   Projects   Framework
                                                       Appendix IV
                                                       Process Criteria

                                                       service needs and mental health problems, planners responded by devel-
                                                       oping additional services. Over time, services such as a medical clinic,
                                                       specialized mental health services, and food services have been added to
                                                       the existing transitional housing and emergency shelter services.

                                                       In addition to flexibility in management, the literature on implementa-
                                                       tion identifies the circumvention of standard operating procedures with
                                                       legal, regulatory, and procedural shortcuts as a strategy for improving
                                                       the viability of a project. It should be noted that there are hazards with
                                                       this’approach related to noncompliance with applicable laws or regula-
                                                       tions causing political or legal pressures. However, it does appear that
                                                       flexibility in project design,, planning, and management in response to
                                                       changes .in external or internal circumstances can enable a partnership
                                                       to take advantage of new opportunities or to address problems as they
                                                       arise. Record reviews and interviews with participants and observers
                                                       are likely sources of data for these measures.

n-,--,,-,         A --
                                                       ning of the partnership project. Indicators and measures for assessing
and Management                                         the success of resource acquisition and quality of management are listed
                         .,       .r
                                                       intable IV.4.

fable IV.4: Process Criteria: Resource
Acquisitiotxand Management                             Indicator                          .I                          Measure
                                                 ‘1‘. Availability     of nonfinancial    resources                   Number, stability, and quality of staff;
                                                           .‘:                  ,I                                    availability of technical resources; amount     of
                                                                                                                      contributed    labor and donated facilities
                                                       Quality-of     nonfinancial    resources                :      Technical and political skills of staff; level of
                                                                                 ,.    ,‘.     *I’     ,,,,           staff training and experience;     extent of staff
                                                                                                                      commitment;     quality of contributed   labor and
              *          “,   ,. ,,         ,_    ,,                          /,                  ,’      ,/       1. donated facilities
                                                       Availability    of financial   resources                       Leveraging ratio; ratio of actual dollars
                                                                                                                      leveraged to the amount expected; timing of
                                                                                                                      receipt of financial resources; stability of
                                                                                                                      funding; use of innovative financing
                                       ,,               Management        of resources                                Use of market analyses and feasibility
                                                                                                                      studies in implementation;        degree of
                                                                                                                      aggregation      of public and private resources;
                                                                                                                      adequacy of financial reporting system;
                                                                                                                      quality of financial recordkeeping;      extent of
                                                                                                                      responsiveness       to funding sources; clarity of
                                                                                                                      responsibilities    of staff; balanced staff teams

                                                        Resourc,es can be either financial or nonfinancial. The availability of
                                                        nonfinancial resources can be measured by the number of staff, quality

                                                        Page 39                                                    GAO/PEMD-90-9     Partnership   Projects   Framework
                                           Appendix IV
                                           Process Criteria

                                           of staff (indirectly measured by staff salaries), and the type and
                                           breadth,of technical resources available through either staff expertise
                                           or contracts with outside experts. In addition, the amount of contributed
                                           labor and facilities is an indication of the ability of the partnership to
                                           marshal nonfinancial resources. These data can be obtained from grant
                                           applications, document reviews, or on-site visits to review files.

                                            The quality of the resources obtained is also relevant, because it is
                                           linked to the utility of the resources, Staff quality measures, such as
                                           stability and technical and political skills, have been related to partner-
                                           ship performance in containing costs. The quality of technical resources,
                                           contributed labor, and facilities should also be assessed,because gaps
                              ;”           here can affect project outcomes. For example, Greater Boston Commu-
         .‘,                               nity Development, Inc., aXprivate nonprofit agency, provided technical
                                           assistance to Inquilinos -Boricanas en Action (Puerto Rican Tenants in
                                           Action) in selecting builders, ‘applying for subsidies, and other matters.
                                           These kinds of technical resources can help partnerships avoid mis-
                                           takes, save time, and accomplish their goals. Again, these data can be
                                           found in written records, by directobservation of performance, or inter-
                                           views with participants and observers.
                         ?                 Successin acquiring financial ,resourcesis most commonly measured
                             .’         ) with .a leveraging ratio. In general, higher leveraging ratios indicate
                                           more successthan do lower ratios. The leveraging ratio can be difficult
                                           to determine because there are multiple layers of leveraging. ,For exam-
                                           ple, the partnership should be,interested in the amount of funds
                             ,’            acquired ,from outside sources relative to the commitment made by the
                                           partners; But the feperal government is more interested in the amount ’
:                                        j of private investment leveraged with a federal grant. Because different
                                        , sponsors are interested in different ratios and because the funds from
                                           these sources are fungible, sorting out the leveraging implications of any
    _’                                     one source can be challenging.
                    ‘,             ,.
                                           However, there are other measures of the availability of financial
               ‘I        ‘,                resources. For example, the leveraging ratio may be high but the amount
                                           of funding available could still be inadequate for the project that the
                                           partnership planned. The ratio of actual dollars leveraged to the amount
                                           expected is a measure of successin obtaining sufficient resources. The
                                           timing of financial resources also is-important. For example, early fund-
                                           ing to cover the initial start-up and operating costs enables partnerships
                                           to formally establish an organization, develop specific strategies, and
                             ..,           line up other.
                                                    I ! resources. In addition, the stability of funding is a measure

                                          ,-Page 40                           GAO/PEMD-90-9   Partnership   Projects   Framework
                         Appendix IV
                         Process Criteria

                         of the continued availability of funds for spin-off projects or as a cush-
                         ion against project delays.

                         Financial and nonfinancial resources have to be managed as well as
                         obtained. One measure of the quality of management of resources is the
                         use of market analyses and feasibility studies to make decisions about
                         the appropriate amount and allocation of resources for different activi-
                         ties. The ability to aggregate resources from different sectors is another
                         management skill needed in partnerships. The acquisition and use of
                         financial resources can be monitored with a financial reporting system.
                         The stability of financial resources, an aspect of their overall availabil-
                         ity, can be.encouraged through responsiveness and accountability to
                         resource providers. These data can be obtained from on-site observation,
                         progress reports to funding agencies, and interviews with knowledgea-
                         ble individuals.

                         Nonfinancial resources, specifically staff, can be managed through clear
                         assignment of tasks and responsibilities. According to Mayer, making
                         divisions of responsibility clear is an important task for executive direc-
                         tors and other lead staff. “The most notable project management
                         problems arose when some major activity fell between the areas for
                         which staffers perceived themselves responsible.” (Mayer, 1984, p. 188)
                         Management recommendations also include balanced staff teams.

Application of Process   definition of case studies is “a method for learning about a complex
Criteria                 instance, based on a comprehensive understanding of that instance
                         obtained by extensive description and analysis of that instance taken as
                         a whole. and in its context.” The case study method may involve on-site
                         interviews with participants, visitsto local neighborhoods, or discus-
                         sions with key actors and community members. In order to provide
                         extensive descriptive data, multiple sources of information and types of
                         data sources are necessary, such as observations over time, participant
                         observation, document review, archival records, and physical informa-
                         tion. (Additional information on the application of case studies and their
                         methodology and benefits can be found in Case Study Evaluations (U.S.
                         General Accounting Office, 1987) and Miles and Huberman, Qualitative
                         Data Analysis (cited in Case Study Evaluations).)

                         Again, it should be pointed out that data on many of the criteria are not
                         readily available, which may hinder future analyses. However, it is pos-
                         sible to obtain certain data rather readily, including whether written

              ‘.         Page 41                             GAO/PEMD-90-9   Partnership   Projects   Framework
 Appendix IV
 Process Criteria

plans exist for leveraging funds, the extent of needs assessmentsor mar-
ket analyses, the stability of partnership membership, the technical
skills of leaders, the extent of public relations efforts, the number of
staff, and so on.

As we have pointed out previously, evaluators need to make decisions
on the measures they use in any given instance based on the availability
of data and the costs of collecting it. In any given evaluation, not all the
process criteria and their associated indicators need to be addressed.
The framework is meant to be comprehensive and all-inclusive of possi-
ble,indicators and measures. It is not intended as a model to be adopted
in its entirety.

Page 42                            GAO/PEMD-90-9   Partnership   Projects   Framework

         Appendix V

         ‘Outcome Criteria

                                                     Housing and community development partnership projects have two
                                                     kinds of effects: direct, tangible effects that can be measured in terms of
                                                     the number of housing units built or the amount of commercial space ’
                                                     developed and indirect, less tangible effects, such as changes in the
                                                     investment potential of a neighborhood. The primary issue in evaluating
     i                                               ,the outcomes of partnership projects is being able to attribute any
                                                     changes-in,    say, numbers of housing units or in investment potential-
                                                     to the partnership project rather than to other interventions or simply
                                                     to the passage of time.

                                                     For both intangible and tangible outcomes, measuring changes and
         Design.Issues                               attributing those changes to the projects may be difficult. First, the part-
                                                     nership process itself is complex, involving many participants and
                                                     requiring a variety of resources.. Second, outside factors, including infla-
                                                     tion, recession, federal or state policy changes, and racial tensions, may
                                                     affect outcomes. Third, many of these effects do not occur immediately
                                                     but develop gradually.

                                                       -The design of the evaluation must include some way to attribute the
                                                        effects measured to the project itself. One way to assess whether the
                                                        project caused the observed outcomes is to compare them to data on
                                                        what would have happened in the absence of the partnership project.
                                                        For example, the neighborhood, or community with the project could be
                                                        compared to one that is similar overall but did not have a partnership
                                                        project. However, it is unlikely that one could find a match close enough
                                                        to allow valid comparison. Comparison of the outcomes of a partnership
                                                        project with those of :projects implemented solely by the private sector
                                 \                 i I Ior the public sector suffers from, the same problem-the    difficulty of
                                           I. :   _s finding projects.in,comparable,contexts.

                                                     ‘An alternative might be to use econometric models to predict what
                                                     would have happened in the community without the project, based on
                      .‘/            /‘I             trends in investment, employment, and other variables. The predicted
-I                          ,’                       outcomes could then be compared to actual outcomes, and gains or
                                                     losses could be attributed to the project. While this is more feasible than
                                                     finding an actual match to the community, econometric modeling is not
                                                     without problems. Models are based on an assumption that explanatory
                                                     variables are independent of one another. They also require the implicit
                                                     assumption of some constant relationships over time (or across regions).
                                                     If these,assumptions are violated, the model becomes less reliable and
                                                     harder to defend. In addition, econometric models may be misspecified
                                                     by omitting important variables or .including extraneous ones.

                                                     Page43                              GAO/PEMDBO-9   Partnership   Projects   Framework
         Appendix V
         Outcome Criteria


         A third design, interrupted time series, allows an inference about what
         would have happened in the absence of the project by analyzing trends
         in the variable of interest over time. For example, if the number of jobs
         in a community remained stable or steadily declined over several years
         and then increased suddenly after the partnership project was com-
         pleted, then there would be some evidence that the project was responsi-
         ble for the increase. However, other plausible explanations for the
         increase in jobs would have to be investigated and ruled out. This design
         has an advantage over econometric models in that it does not require a
         fully specified model incorporating all relevant variables and, thus, does
         not impose the burden of collecting data on all those variables. Its disad-
         vantage is that the analysis does require data on the variable of interest
         for many points in time and the identification of and adjustment for
         time-dependent trends or cycles in the data series. Also, the probable
         delay (or lag) between the project intervention and any observed change
         in the variable of interest decreasesthe strength of the attribution
         unless other possible causes for the change in the variable can be ruled
         A fourth possible design for attributing the outcomes of a partnership
         project to the project itself is the case study. Case studies do not gener-
         ally address what would have happened in the absenceof the project.
         However, sometimes they can build a case for attribution through
         detailed description of project processes and the nature of their link to
         project outcomes. (U.S. General Accounting Office, 1987) For example,
         case study researchers can gather important project details such as the
         timing of funds, delivery mechanisms, and the duration of the project.
         While these details facilitate the building of causal links between the
         project and the outcomes, it is still difficult to sort out the effects from
         the project and those,from other contextual factors.

         Because of the diversity of partnership projects, no one design can be’
         prescribed here. Nonetheless, we emphasize the importance of evaluat-
         ing the outcomes of a partnership project in a manner that maximizes
         the ability to attribute outcomes to the project. Often it will be necessary
         to use several methods, counting on the strengths of one to minimize the
         weaknesses of another.

         We identified three criteria for evaluating the outcomes of a partnership
         project: (1) achievement of intended objectives, (2) other effects, and (3)
         costs of the partnership project. The number of outcome measures listed
         is large. But for any one project, many measures will not apply. The full
         list of measures is intended to encompass outcomes of both housing and

         Page 44                             GAO/PEMD-99-9   Partnership   Projects Framework
                      Appendix V
                      Outcome Criteria

                      community development projects. While the process of these two kinds
                      of projects may be similar, the outcomes are likely to be different. Mea-
                      sures that are not relevant to a particular project clearly should not be
                      used to evaluate its effects.

                      The primary outcomes of a partnership project should relate to the
Achievement of        housing or community development objectives of the partnership. As
Intended Objectives   discussed in the section on process indicators, clearly documented and
                      agreed-upon objectives are <measures of the quality of the planning pro-
                      cess. While the objectives of a partnership may be stated in general
                      terms, such as the preservation of low-income housing in a neighbor-
                      hood, the objectives for any one project implemented by the partnership
                      may be much more specific. For example, objectives for a partnership
                      project might include the rehabilitation of a specific number of housing
                      units or the acquisition and development of a certain amount of com-
                      mercial space.

                      There are two approaches to measuring the extent to which intended
                      objectives have been achieved. Many discussions of partnership projects
                      focus on quantifiable, tangible outcomes, such as the number of housing
                      units constructed or jobs created. A second way to assess the extent to
                      which objectives have been ,achieved is to ascertain the direction and
                      magnitude of changes in the need measures that can be attributed to the
                      partnership project. For example, an increase in the number of housing
                      units relative to the number of households may indicate the degree of
                      success in achieving the objective of increasing the availability of hous-
                      ing. Furthermore, as discussed above, this apparent success would have
                      to be linked to the partnership project in order for the increase to be
                      attributed to, the project. In addition, any look at the change in the rela-
                      tionship between housing units and the number of households must con-
                      sider changes in both sides of the relationship. For example, population
                      changes in the community could also be affecting the relationship.

                      One potential disadvantage. to using need measures for evaluating the
                      effect of a partnership project is that the effects may be small in rela-
                      tion to the need. As a result, decreases in the magnitude of the problem
                      that can be attributed to the project may seem insignificant. In addition,
                      measures of housing and community development need may be affected
                      by many other factors besides the project, such as changes in federal,
                      state, or local tax policies and other exogenous conditions.

                      Page 46                             GAO/PEMD-90-9   Partnership   Projects   Framework
                                                                       Appendii        V
                                                                       Outcome        Criteria

                                                                       Table V.l lists measures of change in the extent of need as well as mea-
                                                                       sures of common outputs. The disadvantages and advantages of, and
                                                                       potential data sources for, the measures previously described under
                                                                       magnitude of need are not repeated here. However, some considerations
                                                                       in applying the other measures are discussed.

Table V.l: Outcome Criteria: Achievement of Intended Objectives
Criterion                         Indicator                                                         Measure
Achievement             of intended     housing          Increase   in housing      availability    Number of housing units constructed     or rehabilitated;    increase in
objectives                                                                                          ratio of existing stock to number of households,       new housing
                                                                                                    permits issued, and housing starts and completions;         decrease in
                                                                                                    number of housing units lost to abandonment,        fire, or demolition
                                                         Increase   in housing      affordability   Decrease in proportion of household income going to rent and in
                                                                                                    interest rates for home mortgages;         increase in rates of
                                                         Increase   in housing      quality         Decrease in housing with inadequate           plumbing, inadequate      heating,
                                                                                                    inadequate      provision for sewage disposal, incomplete kitchen
                                                                                                    facilities, structural problems (e.g., leaking roof, or holes in floors or
                                                                                                    walls), common-area       problems (e.g., broken or missing stairs or no
                                                                                                    working light fixtures in common areas); decrease in housing lacking
                                                                                                    electricity or with electrical deficiencies      or in housing with fire
                                                                                                    hazards, inadequate       light and air, or signs of vermin; decrease in
                                                                                                    average age of housing and in average number of people per room:
                                                                                                    increase in quality of management         of rental units; improvement      in
                                                        /.                                          condition of neighborhood        (abandoned     structures, crime, other
                                                                                                    physical and social conditions)
                                                         Success    in targeting     housing        Extent to which partnership        project   served   targeted   geographic
                                                         benefits                                   area and taraeted’oooulation
                                                                                                                  -       II

Achievement   oftintended                                Relief of economic        distress          Net number of jobs created or retained; quality of jobs created;
community develppment                                                ,_                             ‘decrease in percent of people at or below the poverty level,
objecttves ,.                                     ,,,                                                unemployment       rates, rate of long-term unemployment,        or
                                                                                                    underemployment         rates; increase in per-capita or household income
      a                                                                                             and ,rate of grotith in retail and manufacturing         employment;    increase
                                                                                                    in new capital expenditures        (investment    in new plant and
                                                                                                    equipment);   increasesin amount of retail sales, amount of service
                  ,‘,            ,‘(    !                                            !              receipts (income from the service sector), or wholesale trade; square
             ;,              :     1:                                                               feet of,commercial       space constructed     or rehabilitated;  change in
                                                                                                    number and type of businesses;           decrease in migration of population
             ”          /                               ,’                                          and businesses;       number of busrnesses assisted; decrease in crime
                        .,              2                 ”     \ ,’                                rate.by crime type
                                                         Relief of physical   distress              Decrease in extent of garbage-littered     streets, unpaved or broken
                                                                                                    streets, cracked or broken sidewalks, inadequate       drainage and
                                                                                                    sewage facilities; decrease in number and concentration        of
                                                                                                    condemned      or abandoned    buildings; decrease in percentage   of
                                                                                                    streetlights  missing or ineffective
                                                        Success in targeting community              Extent to which partnership project served targeted geographic
                                                        develobment   benefits                      area or taraeted oooulation

                                                                       Many descriptions of “successful” partnership projects do not relate
                                                                       achievements to magnitude and distribution of need. Instead, the num-
                                                                       ber of housing units constructed or jobs created are cited as evidence of

                                                                       Page 46                                                 GAO/PEMD-90-9     Partnership    Projects   Framework
Appendix V
Outcome Criteria

success. But these are only partial measures of the success of a project.
For example, in housing, both the quality of the new units and their
affordability for neighborhood residents and low- and moderate-income
people may be other aspects of whether the intended objectives have
been achieved.

Table V.l also shows targeting success as an indicator of the achieve-
ment of intended objectives. Success in targeting means that a partner-
ship project is effectively reaching its intended geographic and
demographic targets.

Measurement along both the places and people dimensions of targeting
is necessary to assess the overall success in addressing the needs of a
particular area or population. For example, success in geographic target-
ing may not be sufficient if the population of the area has been dis-
placed. Thus, in the case of a housing project that improves housing for
its geographic target of a low-income neighborhood, another measure of
targeting success could be for the low-income population and would
include ,the number of low-income residents in the improved housing

Similarly, in community development, the number of new jobs created is
insufficient as a measure of success. The quality of jobs is an important,
often-neglected dimension of job creation. Job quality can be a function
of pay, skill level required, and opportunity for advancement and of
whether a job is full- or part-time, temporary or permanent. It is rele-
vant to assessing any change in unemployment or underemployment
that may be attributed ,to the partnership hroject.

 Another concern with using job creation .as ,a measure of effectiveness is
 the possibility that some.jobs,:inay have been lost through modernization
 o.f equipment or displacement of jobs from one area to another. Because
 of these possibilities we recommend the use of net jobs created or
 retained (that is, number of jobs created or existing jobs retained less
jobs lost) rather than gross jobs created. Net job creation here refers to
job,creation in the project area; not in the national economy as a whole.
 Information on job quality and job creation can be obtained through
 state and local employment service records or surveys of local busi-
nesses involved in the project.

A consideration in evaluating success in achieving intended outcomes is
the durability of those outcomes. Thus, for housing affordability, one
would be concerned about whether the newly affordable units remained

Page 47                            GAO/PEMD-90-9    Partnership   Projects   Framework
          Appendix V
          Outcome Criteria

          affordable to lower-income families over time. The management of new
          or rehabilitated rental units becomes important in the maintenance of
          the improvements. Similarly, as mentioned above, the permanence of the
          new jobs is a factor in their quality. All these suggest the desirability of
          follow-up evaluation work on the long-term effects. But while relevant
          to the evaluation of outcomes, an examination,of the duration of such
          changes would add to the cost of an evaluation, since it means gathering
          data at multiple points in time. In addition, many factors affecting hous-
          ing and community development are likely to change, making the link to
          the project tenuous and difficult to evaluate.

          Targeting success is an indicator of achievement of intended community
           development objectives, just as it is for intended housing objectives.
           Commercial development projects provide an illustration with a mix of
          targeting goals. For a project that assisted businesses, measures of suc-
           cess might include not only the number of new businesses started and
          jobs created but also the number and type of businesses displaced, the
          match of the new businesses to the needs inthe community, and the
          extent to which new jobs are filled by low-income and unemployed
          residents of the area or new businesses started by local residents.

          The methods for gathering information on targeting success are similar
          to those for determining the distribution of a need. However, census
          data .are not relevant unless new data are available after the project was
          implemented and completed. The household directories compiled by pri-
          vate firms, community surveys, and surveys of local businesses and
          housing providers .could provide more current, but more costly,

                    ‘. ;
          In additionmto the intended outcomes mentioned above, table V.2 cites
          other effects of ljartnership projects, either unintended or secondary, to
          the purposes of the partnership project. We have categorized these as
          effects on (1) the public sector, (2) the private sector, (3) community
          residents, and (4) the partnership organization itself. Several of these
          measures suffer from measurement difficulties because the data may be
          sensitive or difficult to obtain. However, rather than ignore these
          effects, we have listed them as an indication of the full range of effects
          that a partnership project can have. Some effects may stem specifically
          from the partnership aspect of the project, while others might be the
          result .of any housing or community development project. In addition,
          most of the variables listed can be affected either positively or nega-
          tively by the project. The attribution of these effects to the project will

.’   ;.
          Page 48                            GAO/PEMD-90-9   Partnership   Projects   Framework
                                                        Appendix     V
                                                        Outcome     Criteria

                                                        be more difficult than for measures listed under the achievement of
                                                        intended objectives because, in general, the effects tend to be less tangi-
                                                        ble and the link between the project and the effects less direct.

Table V.2: Outcomes: Other Effects
Criterion                        Indicator                                            Measure
Effects   on public sector               Changes in state or local                    Changes in state or local use of federal assistance, administrative
                                         government  program activities               procedures,    authority over and accountability     for projects,
                                                                                      administrative   costs, use of private-sector    expertise and financial
                                                                                      resources, or local agency relationships      and coordination     activities
                                         Changes     in political power base          Changes     in relationships   with neinhborhoods        or with private      sector
Effects   on private   sector            Financial   returns on investment            Profits or revenues;     changes   in tax liability or operating      costs
                                         Extent of spinoff   development              Number     of new or expanded businesses or new development
                                                                                      projects   initiated after completion of partnership project
                                         Relationships    with local                  Changes in public image of private sector partners; changes in
                                         government      and neighborhood             nature and extent of participation  in local development decisions                 or
                                         mows                                         in other partnerships with local aovernment
Effects   on community       residents   Changes in self-determination          of    Changes in political participation    by local residents or their
                                         local residents                              involvement in neighborhood       development    activities or
                                                                                      organizational involvement
                                         Changes     in community        as a place   Changes in the costs of doing business, employee stability and
                                         to invest                                    satisfaction, purchasing power, market opportunities; nature and
                                                                                      extent of spinoff development
                                         Changes in neighborhood                      Changes in appearance   of neighborhood,            crime rates, or retail and
                                         environment                                  commercial choices available to residents
Effects   on partnership                 Changes     in capacity to plan,             Changes in number and amount of private sector contributions        or
                                         manage,     and finance projects             contacts with other development      organizations;  extent of new
                                                                                      resources obtained;.changes     in staff quality and number of new
                                                                                      staff hired; changes in scale or complexity of activities, stock of
                                                                                      caoital assets. or flow of revenues and exoenditures

Effects on the,Public                                  Local government participation in partnership projects may change
Sector               ‘.                                otheraspects of their activities. For example, increased cooperation
                                                       with the private sector may lead to a reduction in local dependence on
                                                       federal assistance. The public sector may simplify its regulations in
                                                       order to facilitate development activities by private-sector entities. For
                                                       example, zoning and land use laws that restricted potential business or
                                                       housing development could be adjusted to encourage private-sector
                                                       involvement in a partnership project. However, as discussed above,
                                                       working with the private sector may also result in decreased authority
                                                       over projects, with a potential for diminished public accountability.

                                                       Data for assessingchanges in local government activities may be availa-
                                                       ble from local government records and documents. For example, changes
                                                       in regulations may be determined through a review of public documents.

                                                       Page 49                                                GAO/PEMD-90-9      Partnership     Projects    Framework
                                                       Appendix V
                                                       Outcome Criteria

                                                        However, decisions about whether any changes in regulations have
                                                        resulted in simplification and in fact stem from the partnership project
                                                        have to be based on comparison to the previous regulations and on the
                                                       judgment of the evaluator. That judgment may be informed by inter-
                                                       views with experts or those who are affected by the regulations.
                                                       Partnerships that address housing and community development needs
                                                       may also lead to increased use of private-sector expertise and financial
                                                       resources in responding to other social problems. Moreover, money from
                                                       the .repayment of loans to the private sector to encourage housing and
                                                       community development projects may be used for other local projects.
                 ,’                                    Information on these changes in funding arrangements may be available
                                                       from local government accounting records and management information

                                                        The potential effects above may result from the involvement of the pub-
                                                        lit sector with the private sector in partnership arrangements. However,
                                                        secondary effects on the public sector can occur from housing or com-
                                                        munity development projects, even if they are not partnership ventures.
                                                        Specifically, the degree of success of any housing and community devel-
                                                       sopmentproject can affect the public sector’s relationships with commu-
                                                        nity residents. If a partnership ‘project is expensive, unpopular, delayed,
                                                        or unsuccessful, a local government may lose support for other activi-
                                                        ties. In contrast, a successful partnership project may increase local
                     ,’                                 interest in future projects as well as the popularity of the local govern-
                                  ,, ,‘:          .,   ‘merit. Potential.,,sources of information on these changes are local media
                                I/j.                    reports, and community surveys.

Effe&i     on’,the        kJiv&&            ‘:’       ‘:Private;sector partners in a partnership project, may represent either
                  ,’ ,/ ..: _,: 2: ::               1, .for-profit or nonprofit organizations. In either situation, if they made an
Se&f? ‘: !‘i ‘,’ Ii ,: : j                              investment in the project; one of the effects may be profits or other
                                                    . ‘financial’benefits from’participation in the project. The amount of prof-
                :           ‘t     ;:                   its is,one measure of.the financial returns of participation in the part-
                  “ ‘.         /I.                      nership to the privateesector. Private partners may also benefit through
                                                        changes in their tax liability. For example, some partners in low-income
                                                        housing projects have been able to use the Low Income Housing Tax
                                                        Credit. An additional financial benefit may be lower operating costs,
                                                        depending on the nature of the project and any incentives that may be
                                                        offered by the public sector. Tax records, accounting records, or surveys
                                                        are potential sources of information on the profits and tax liability of
                                                  >. the private sector. Of these, tax and accounting records may be difficult
                                 ‘.                     to obtain ,becauseof confidentiality issues.

                           /I          .,

                                                       Page 60                             GAO/PEMD-99-9   Partnership   Projects   Framework’


                                 Appendix V
                                 Outcome Criteria

                                Partnership projects may stimulate different kinds of spinoff develop-
                                ment in the private sector. Other private sector organizations may be
                                attracted to an area as a result of a local partnership project. For exam-
                                ple, projects that expand the commercial base can create other jobs
                                because of demand for housing, retail, and other services by the employ-
                                ees. This spinoff development can be measured by the number of new
i                               businesses that are initiated or expanded after the initial partnership
                                project is completed. In addition, the private sector may support part-
                                nership projects that address other problems or target other areas.
                                While spinoff development can be an important side-effect of partner-
                                ship projects, it may be costly to measure and difficult to attribute to
                                the project, because of the delay between the completion of the project
                                and the initiation of related development.

                                Another indicator of effects on the private sector is changes in relation-
                                ships with the local government and community. The public image of a
                                private organization may improve if the partnership is successful and
                                the participation of the private entity is publicized. The nature of the
                                private sector’s relationship with the local government can be measured
                                by any changes in the extentof private-sector participation in local
                                development decisions and subsequent partnership projects. Data
                                sources for these measures include local government planning docu-
                                ments and surveys of government officials and business executives.

         Effects on Community    Effects on the community that might not be the main focus of the part-
         Residents              nership but could occur as a result of a partnership project include the
                                self-determination of community residents, changes in the community as
                                a place to invest, and changes in the neighborhood environment. The
                                self-determination or “empowerment”       of local residents is a potential
                                side-effect of partnership projects that involve residents as partners.
                                 “Self-determination”   refers to the development of local leaders and the
                                increased involvement in political and development activities by
                                residents. In measuring self-determination,    it is important to address the
                                issue of whetherchanges in local political participation stem from a
                                change in the type of resident (for example, if lower-income residents
                                are displaced by higher-income residents) or to actual changes in the
                                involvement of the targeted population.

                                The results of a partnership project may also change the community as a
                                place to invest. For example, if infrastructure services (such as roads
                                and transportation)   are improved, the costs of doing business in an area
                                may decrease. If commercial development were to result in more

                                Page 61                             GAO/PEMD-90-9   Partnership   Projects   Framework
                            Appendix V                                                                              I
                            Outcome Criteria

                            employment in a neighborhood, then employee stability and satisfaction
                            may increase along with the purchasing power of the community and
                            new market opportunities. These changes in the community may
                            encourage spinoff development. Spinoff development occurs when com-
                            mercial interests are attracted to an area that has been the focus of a
                            public-private partnership project. Another form of spinoff develop-
                            ment is when additional development projects are initiated\in a commu-
                            nity, perhaps in a different neighborhood. Information on changes in the
                            community as a place to invest may be available from local business
                            license records, employment records, or a survey of local businesses.

                            Environmental changes such as esthetic improvements, reduced crime,
                            and new retail and commercial choices are other potential secondary
                            effects from a partnership project. They can be measured through com-
                            munity surveys and direct observation, as well as through the measures
                            discussed under housing quality and physical distress. As neighbor-
                            hoods are improved, they may become more attractive places in which
                            to live and invest. This can result in gentrification, or the displacement
                            of low- and moderate-income residents with higher-income households.
                            The costs associated with displacement and gentrification are discussed
                            below under costs to the community.

Effects on Partnershin
                             The major potential effect of public-private partnerships on the partner-
Organization                 ship organization itself is its development as an independent organiza-
                            tion. This may occur as the partnership gains experience in planning and
                            managing partnership projects. Specifically, the partnership organiza-
                            tion can improve its capacity to plan and manage new projects by devel-
                            oping contacts and acquiring contributions from other development
        .                   organizations. Measures of the number and amount of these contacts
                            and contributions can be examined. Also relevant are staff changes in
                            terms of quality of number of new staff hired. As discussed above, the
                            skills of the partnership’s staff and management are linked to successful
                            fund-raising and implementation of local projects. Changes in the scale
                            or complexity of activities, the stock of capital assets, or the flow of
                            revenues and expenditures are also pertinent. These measures provide a
                            means for assessing the potential of the partnership organization to
                            undertake future ventures. Data on changes in the partnership’s capac-
                            ity can be gathered through reviews of annual reports and other organi-
                            zational documents and records.

                            Page 62                            GAO/PEMD-90-9   Partnership   Projects   Framework
                                                                                                         Appendix V
                                                                                                         Outcome Criteria

                                                                                                    Just as the partnership may have effects on the public sector, commu-
Costs of,the,                                                                                       nity residents, the private sector, and the partnership organization, so
Partnership’Project                                                                                 may costs of the partnership accrue to these groups. The costs of the
                                                                                                    partnership project may be financial, political, or social. (See table V.3.)
               /                                                                                    Financial costs. include accounting and opportunity costs. Accounting
                                                                                                    costs are’the amount of resources that each sector has invested in the
                                                                                                    project and the risk involved in that investment. Opportunity costs are
                                                                                                    the value of alternative purposes for which an investment could have
                                                     ,.                                             been used. Measurement of the costs of partnership projects is impor-
                                                                                                    ,tant’as a basis for determining the cost-effectiveness of the project. In
                                           i 1’                             ,/                      combination with data on the effect of the project, cost per unit of hous-
                                   I          ;’                                                    ing or perjob created can be estimated. This informs a comparison of
                                                                                 ‘.              I the :effectiveness of the project to other kinds of interventions. While
                                           ‘.                                                     : important, data on some of, these costs may be difficult to obtain
                                                ,”                                             :, because of ,the sensitivity of the information.
                                                                                                               ,.;             ),.

Table V.3: Outcomes: Costs of the Partnership Project                                                                                                 ,          1’         ..           ,’
                                 indicator                                                                                                                  Measure
Costs to public sector.                                                     Accountingcosts                           ”                          ‘/        Financial, risk of, participation in project; nature and amount of
        ,I,      ,‘,,,                                                     ,‘.        h,‘,,,.                                                              ,investment; changes in revenues from use of tax increment
                                                                                                                                                           financing and other financing strategies
    ‘,   _)        ,/                  ;
                                                                            Cpportunity     costs                             .’        !                  Amount of ihvestmeht; changes in revenues from use of tax
                                                                              .!_       ,-.                                                                increment financing and other financing strategies; social value of
                                                                                   ”          ’                                                            forgone investments
                                                                            Political costs                                                                Change     in authority       and accountability      over projects
Cost toprivate           sector                                           : Accounting, costs                                                              iVat,ure and amount of investment; financial risk of participation                      in
                                                                                                                                                           the project; investments  made below normal size threshold
          ,I       .‘,
                                                                          “‘Opportunity                  costs                                             Alternat@   return on ‘investment             such as the money market interest
                                                                                                                                                           rate      ’ : ‘“,I:; ._
                                                                           Political costs                                                                 Change     in authority       over projects
Costs to community residents                                               Accounting costs                                                                 Nature and amount of investment including nonfinancial resources;
        ,,        ,! :;,,    ,;,                                      ;    8.1 (‘, ,,“‘1,  *                      :                 , :                    ,f/nancia! r]sk.of participation     in the project; .costs,of new units to rent,
   ,,    ‘1                                                                                                                                                 or buy; c,han’ge’in property taxes; moving andreloc’atron’costs             of’: f,
                          .1                                                              ‘.       ,‘,    1               ;             ,:                                                                            i’ , I.#,
                                                                                                                                                          “displaced’residents’and         businesses                      / ,I :, ( r’ ‘,!
            .I     ,’ :                                                   :,So&i&costs~l’                                                    ”        :    Change in social networks            for displaced      residents;      number     of jobs
                    ,:.:!          ,,                                       ...  .,.(’                           .,                                        displaced !j,S.   ,‘:
Costs to p,artnership                                         ;            Account/rig   costs                                     .,                      Capital costs; staff salaries; value and depreciation of physical
                                                                                      >,                                                                   equipment,and    faci@es; cost of fundraising and planning
                            ,,>,           ,              ’                                                                                                          .,/’        ,.,:.

Potential Costs to the                                                                                   Accounting costs to the public sector can be measured by the nature and
Publi@$t&-                                                                                               amount of investment ,and the financial risk involved in participation in
 ‘!                                                                                                      the .project The nature of ,the investment is an important measure,
               L                                                                               :         because.the public.sector may provide staff, land, facilities, and other
                                                                  ’        I’                             /_       !   ,I          I;I      : ‘i,,

                                                                                      (              L Page 63                                                                           GAO/PEMDQO-9         Partnership       Projects    Framework
                                                        Appendix V
                                                        Outcome Criteria

                                                                         nor-monetary investments. Their financial value has to be estimated in
                                                                         order’ to determine the total investment of the public sector. In addition,
                                                                         the public sector may have’used,financial tools to offer incentives to the
                                                                         private sector. Specifically, the use,of tax increment financing, deferred
              _, ,~                                                       loan repayments, and discounted interest rates involve financial costs to
                                                                         the public sector. Data on these costs should be available in local govern-
                                              :. ‘,                       merit records.
                                              ;                 ,,                       :                     !
             /’            >.                                            Opportunity ‘costsare an indicator ‘of the costs of a partnership project
                        ,’                       ~             ,,. to the public sector. Opl5ortunity costs reflect the fact that resources are
                    ; ‘I.                            ,‘.            ,’ limited.<Therefore; anydecision to invest resources in a particular pro-
           ,I., ,, .’        ;;;> .”                       :             ject implies that other uses of those resources have been forgone. In
                                          ,.                      ~ ” -other words, .o$ortunity costsimply a choice between different public
                              _I ,               ,             ;’         goods.%or example;opportunity costs occur when a local government
                                                             <            decides to fund a:partnership project rather than providing more of
-        I.’ ,/, > ,,i ‘.
                                                                          some alternative,service. While this kind of opportunity cost can be
 1       G                                 I,, ‘, -, ‘i, :jj-&~&d :ih $&‘-&‘of t;fie;a&G,& ,6f .ifiv&the$ ,,tKe,social value ‘of the
                                                                          forgone services is difficult,;t~lestimate.               ” ~ ; ’ ‘I 8. 8 : : : _
                                                                                         ,. _L ”i 1
                  ,.            ‘: 3. /‘,, ,I.’                       !>A$otential political cost to the public sector is the loss of: public
                                  :, ;                                :‘.accountability         for projects. While the blurring of responsibility may
                      :, ‘,I                             ‘!          , make ,the partnership more -flexible in ,responding to changes in circum-
                                       I,                                 stances, it also .opensthe door to potential mismanagement. In addition,
                                                                         if a project.is not well-received in the community or if community expec-
                               ;      :,‘/ : :                       ’ tations for a’project are’not met, ,the public sector may lose overall sup-
          , .-1                    ; jl, ,, / 8. ,, ‘, port @‘Well- as,support for other projects. Methods for measuring the
            ,, “,, .i                           ‘.,
                        , .,          ,;,                    ‘, .‘, ,; loss inauthority’ and accountability,,over, projects were discussed in
                                                                        ’ appendix Iv oniprocess criteria.              :
                     .”                             : >‘,,,, I” ,’ /, ;: :, ‘,, >’,. Q,C’,J       :,                  :: )_>S,,.
       :I,,-;./$;,,, :.,,*
                        +,,,),(, I.,,($ .,‘.!‘,,I,/            (,           ‘,.,‘, .,, k:/, ,, “.. 4 j            .I .(,.,. ..I .,,     ..L..      .,,. 8
     &t&&i              cgeti ;t; tfib, .‘? y, :’ The ,accountingcosts to the private sector are the amount of private
     Private                                     r,:: ,,,,-.,;. -investment and’ihe: financial risk of participation in the project. Another
          ,::.: i Sector
                     /-;- : .,73’.,,,,
                                    ” t                                  financial costmay occur .if a private sector’,organization makes an
                                                                         investment beloiv its normal size threshold because it costs more per
I                              ,’ .” ’                             “‘,’ ‘dollar to,pro%ess:For example, somefinancial institutionsordinarily             j.
                                                                         .Wouldnot handle smalldevelopment loans but might do so<as,part of a .,
                                                                         public-private partnership because of the good will engendered through
                                                                         participation in a community effort. The costs to the private sector,
             (.                                                          include;opportunity ‘costs:;If a.for-profit comppy makes,an investment
                                                                         a%Gth     a rate :of zreturn lower than the expected rate of return. for other,,
                    “,                       “./                         investments (measured; for example, by the money market interest rate
                                  (’ ,,.           ,.,“,‘,           I. ‘or the Standard,arid Poor ,index,ofstock prices), it has incurred an
                      ’                                                  ofiportunity ‘cost; Finally, like the public sector, the private sector may

      ‘/ 3I          1. ,..;,      ; /. ‘,;    ,‘;. .,; Page   54                                   ” GAO/PEMD-SO-9     Partnership   Projects   Framework
                           -..-_                             ----_..”      _...__.._

                                         Appendix V
                                         Outcome Criteria

                                        also experience a political cost in the loss of authority over projects. For
                                        example, in housing, the private developer may concede some authority
                                        over the price of new housing units in order to gain public sector
                                        involvement in a project.

Potential Costs to                      Community residents can incur both financial and social costs. Financial
Community Residents                     costs include the amount of investment by community residents and the
                                        financial risk of participation in the project. The community may invest
                                        in a partnership with volunteer time, “sweat equity,” and other
                                        resources in addition to money. Information on the investment that com-
                                        munity residents have made may be available in project records. Other
                                        financial costs are the costs of new housing units to rent or to buy and
                                        changes in property taxes. For example, in Baltimore, residents of neigh-
                                        borhoods adjacent to the Inner Harbor area that was developed through
                                        a public-private partnership project found that tax assessments were,
                                        rising along with the value of their property. Some of these residents
                                        were on fixed incomes and were confronted with an increased tax liabil-
                                        ity, although their wealth was increased. If local businesses and
                                        residents are displaced.from a neighborhood that has been improved by
                                        a public-private partnership, then relocation is another financial cost of
                                        the project.

                                        Displacement may also result in social costs. If a project displaces local
                                        residents through gentrification or commercial development, the dis-
                                        placed residents may lose not only their homes but also social connec-
                                        tions to their neighbors, local businesses, and services. Jobs can be lost
                                        because of relocation of businesses, The measurement of these unin-
                                        tended costs of partnership projects provides a more complete picture of
                                        their overall effectiveness in achieving housing and community develop-
                                        ment objectives. However, both the social and. financial costs of dis-     ,
                                        placement may be difficult to estimate because the primary source of
                                        information is the displaced residents, who may be dispersed and diffi-
                                        cult to trace; even if they were located, it might be difficult to collect
                                        from them the data necessary for this measure.

Potential Costs to the                  Many of the costs to the partnership have been discussed as they relate
Partnership                             to the public and private sectors. However, some costs may fall on the
                                        partnership as an entity, apart from the member organizations. These
                                        may include capital costs (for example, interest to be paid on borrowed
                                        funds), staff salaries, the value and depreciation of physical equipment
                                        and facilities, and other project outlays. These should be measured in

           ,.“.   ‘.,,‘,           .)   Page 66                            ,GAO/PEMD-909   Partnership   Projects   Framework
                                                                                     Appendix V
                                                                                     Outcome Criteria

                                                                                     ‘order to assessthe costs to the project itself, as opposed to the costs
                                                                                     incurred by any of the participating organizations. This information
                                                                                     may be available from project records or government tax records.




                             :       i.’                 /,

                                                                                               :   ’

                                                                                                                      !   .,        ,.:
                                       ,                               ,,                                                      _’

      .,‘,                                                    ‘.            ,,

             .’         <_

..,               ‘,I                                J                                                 ,,   .:   ‘,

                                           ,.i                                   1

                                                                                     Page 56                                              GAO/PEMD-90-9   Partnership   Projects   Framework
     <Appendix VI

 : Evaluating Pubprivate                                                            Partnerships From a
; Federal Perspective

                    ‘,                                 ’       In this section, we move from a discussion of the evaluation criteria,
                                                               indicators, and measures applicable to specific public-private partner-
               ’         ‘.                                    ship projects to a consideration of the federal programs’that support
                                                               those projects. The issues involved are quite different, although in some
                                                               cases the differences primarily concern meeting the data needs of evalu
                                                             ~ating multiple projects as opposed to single cases. Thus, while there is
 /                                                             some overlap with the framework presented in the previous appendixes
-1                                                             (particularly in assessing the need criteria), the approach here is quite
                                                               different. We identify four broad federal questions about public-private.
                                                               partnerships in housing and community development and discuss the
                                                               data needs associated with each one. As indicated earlier, not all the,
                                                               measures and analyses discussed here will be needed for every evalua-
                                                               tion of federalsupport   for public-private partnerships; which ones are
                                                               appropriate will depend on the specific program under review and the
                                                               purposes of the evaluation.
                                                               There is no one federal program with the direct objective of supporting
                                                               publ&private     housing and community development partnerships.
                                            .,             : Rather, a number of federal programs support projects operated by such
                                                               partnerships when the activities of a project are considered consistent
         ,I                                                 ‘I with the purpose of the federal. program. This section presents a general
                                                 ._            framework intended. to be adapted for use in evaluating public-private
                                                               partnerships within a variety of program contexts.

                                                                   In addition, the framework is designed to facilitate evaluations involving
                                                                   several: different types of comparison. First, the framework could be
                              .,         ., .;/..                  applied to evaluate the .use of public-private partnerships by a single
                                               .;,      _i ,:.:>federal program.Second, the-evaluation could focus on the partnership
                                                 ::. ,,      z’sI <,mechanism, regardless of,the,specific federal program providing sup-
                                                                  .port. In this case, the crucial issue might be how well different partner-
                                           )’        I,,m ,.. .:!,shiparrangements       succeed; Third j the evaluation could be designed to ,:
                                                      ., ,,,       look across programs to determine whether partnerships are’more~likely ”
                                          ‘..           ‘.       * to succeed in some program contexts than in others. Fourth, the frame-
                                        ,_ ,”
                                                            -s; 1 work could be used to compare partnerships to other forms of program
                                           ..(           _..       delivery, either for one.program or across programs,

                                   ,’            :,,
                                                            Page 57                               GAO/PEMDBO-9   Partnership   Projects Framework
                                                 Appendix Vl
                                                 Evaluating Public-Private    Partnerships    From
                                                 a Federal Perspective

                                                 As we reported earlier, we were unable to determine either the number
        What Federal                             of partnership projects or the amounts of federal funds allocated to
        Resources Are                            them for the 46 federal programs we identified as providing support to
        Allocated to Support                     housing and community development partnerships. (US. General
                                                 Accounting Office, September 1989) We could find no federal or private
        Public-Pkiv&                             data base that provides this information for these programs. Of course,
        Partnership Projects                     individual agencies or federal program staff could collect such informa-
        in .Housing and                          tion for the programs under their jurisdictions.
        Coqunity                                 It would be possible to develop a data collection instrument to classify
        Development?                             projects assisted under these 46 programs into those that did and those
                                                 that did not involve public-private partnerships. This could be done once
                                                 or continually and could involve the collection of additional information
                                                 describing the partnership arrangements. While such data collection
                                                 could be costly, savings could be realized if a representative sample of
-                                                projects were used to derive estimates.
    1                                            The kindsof measures for which data could be collected are indicated in
                                                 table. VII. These measures are essentially descriptive, but they are
--1                                              important as benchmarks against which to carry out analyses, including
                                                 those involving comparisons,of the relative effectiveness of different
                                                 programs, with similar goals. For. example, an evaluation of partnerships
                                                 supported by HUD'S housing development grant program could include
                                                 an estimate of the amount of federal funds invested per unit of housing
                                                 constructed. Depending on the purposes of the evaluation, this ratio

-1                        ‘,
                            ’   /.:’
                                                 might be compared to the per unit costs of construction for projects car-
                                                 ried out entirely in the,public sector or to those supported under a dif-
                                                 ferent program;suchs as HUD'S mortgage insurance program for
                                                 moderate-income rental and cooperative housing.
        Table WI: Federal. Support for Public-
        Private p+n@ips,     ,            j      Indicpjpr,,                  ,, /       ,:    :j       Measure
                    ,‘!,                         Use,of partnerships                                    Number of partnership projects supported;
                                                   :I                                          I        partnership projects as proportion of all
                                                                                                        I   ,

?                                                Financial and nonfinancial   support   for            Types of support offered under program and
                                                 partnerships                                          number of projects supported      by each type;
                                                                                                       obligations to and outlays for public-private
                                                                                                       partnership  projects through program;
                                                                                                       partnership obligations and outlays as a
                                                                                                       proportion of total obligations and outlay for
                                                                                                       program; dollar value of staff time devoted to
                                                                                                       assisting partnership    oroiects

                    ‘.                           Page 58                                             GAO/PEMDBO-9    Partnership   Projects   Framework
                                                Appendix VI
                                                Evaluating Public-Private   Partnerships   From
                                                a Federal Perspective

                                                As we note in table VI.1, federal support for public-private partnerships
                                                can be financial(grants, loans, or tax incentives) or nonfinancial (regu-
                                                lations or relief from them, technical assistance, or managerial, advice).
                                                For any given program, anevaluation should, to,:the extent feasible,,
                                                include an analysis of the types of assistance provided by the. program
                                                and the number of projects actually supported through that type of
                                                assistance. Such information would permit an assessment of the relative,
                                                efficiency and effectiveness of different forms of support.

                                                Returning to the ,housing comparison, an analogous evaluation might
                                                compare the:number of housing units constructed with project grants
                                                under ~thehousing development .grants program to the number built with
                                                loan guarantees under the mortgage insurance program. Such a compari-
                                                son could take into account the effect on low- and moderate-income
                                                households. For examQle; a mortgage insurance program could build
                                                more housing’units than a comparable grant program, but it probably
                                                would not assist low- and moderate-income households to the same
                                                degree becausehigher-income,households are also likely to benefit from
                                                loan guarantees.

                                                  In the case ,of financial assistance, the appropriate measures of the mag-
                                                  nitude of support are budgetary, focusing on the obligations (generally
                                  ‘,              inthe form of grants or contracts) made for, projects and the actual out-
                                                  k&expenditures)         from federal funds directed to those projects. We
                         I                    : found that these data are not readily available in the summary tables of
           :             i’ :                  ‘, the ,annual budget prepared by the U.S. Office of Management and
                                                  Budget., This suggests that for each program, a detailed analysis would
               .;        .I            .,        ,‘~~~~,‘to’be’c?~ductedin order to determine the extent to which grantees
                                          !.. (or “‘mediating agents’,:,)direct.a#era1 funds       ~.’ .‘,i to
                                                                                                            ! public-private
                                              ” partnershipsi :                       :,:‘.‘.‘, ,;
    ,i ‘, : ,‘:y                ,1,        ‘: : , (( .., ,, , I               ” ,I                                           :
                                                  $+nfina@al suljport may be difficult to quantify. But one possible’meai
                                                  sure ‘is the’ dollar value of the time federal staff spend in providing tech-
                                                  nical, planning,, or management assistance. Agency records normally
                                                  permit estimates of time and total compensation costs for personnel
                                                  engaged in providing such assistance.

                                                Page 69                                           GAO/PEMD-90-9   Partnership   Projects   Framework
                                                                                                Appendix VI
                                                                                                Evaluating Public-Private           Partnerships   From
                                                                                                a Federal Perspective

                                                                                                As we discuss in Partnership Projects (U.S. General Accounting Office,
What .NeedsAre :                                                                                September 1989), public-private partnerships address a variety of hous-
Addressed by                                                                                    ing and, community development needs. In table III. 1, we have provided
Federally Assisted                                                                              a framework for considering, the magnitude of housing and community
                                                                                                development needs addressed by partnership projects. The measures
-                                                                                               associated with those’ needs consist of a variety of community charac-
Partnerships?                                                                                   teristics, national data on many of ‘which are available through the
                                                                                                debennial census and the biennial American Housing Survey or through
                                                                                                the aggregation of locally collected data. Table VI.2 provides indicators
                        .i                    /a’                                               and measuresof the need for public-private partnerships in housing and
                                    :,          .;                     3                        community;development.      We will not repeat the detailed discussion of
                                                                   -’ ,/                        data sources ,for these measures’ that is given in connection with table
                  .:,                                          ! ., ‘; (’                       111.1.         :            ” I,
                             :_              ;                 _,         /                                                ),

Table’Vl.2: Need foi Public-Private.Partnwkhips                                                       in Housing arid CommunitycDevelopment
Criterion “’          ,:     ‘,    .Indicator                                                            : ). ”         ‘Measure      ‘8
Magnitude                    of :housing                 needs                   Extent to which, housing.is!        not        aIRatioSof ‘existing.stock   to number of households;            number     of housing
                                                                                 available                                        starts; rates of :hpusehold formation
                                                                                 Extent to which housing is not                   Proportion of household income going to rent; rate of
                                                                                 affordable                                       homeownership;    sales price in relation to household income;                 interest
       ,:                                    I’, ,;“..                           .,::,           ,“?    ,’    ,’                  rates for home mortgages;    ratio of shelter beds to homeless
            _’               !.                                ,I’ :/“/.          :     _.   !,                                  ,population:.
              .‘1                        :                                       Extent to which:housing          is of poor.         f3tent of, housingrwith        inadequate       plumbing, inadequate      sewage
                                                                             ”   quality                                              disposal, structural problems (e.g., leaking roofing or holes in floors
                         ;                                             ,/I            :             I                      .,’ /(” or walls), or~tiithdommon-area                problems (e.g., broken or missing
                                                                                   1,. ,.           ,,‘,i            : ,_,/.        ,stairs,pr n,o working.light        fixtures); inadequate       heating, lack of
                                                         ‘,:         l,‘,                       $/.,              .’       !        : electrjcity or electrical deficiencies,          fire hazards, inadequate    light
                                  :I                                                                                                  and air, or signsof vermin; age of housing; extent of overcrowded
                                  ‘:,              ”              :, ,;.,:
                                                                   ‘.               ,’ ?                                       ” ‘,. ~housing;lqualitylofL       management        of rental units; condition of
  ;,        ‘,/          ‘1,; *:: ;        ” ,j.,_I ,i: !; 0.. ,,;: “., ,; ,‘/$. /,   :‘j                      2.:::        .:,,; ,?!~~~.h~~rh?9~:(ab~n,~oned               structures, crime, other physical and
                                                                                                                .,                    socral condttrons)’        ”
                                         :   ,,’ ,‘,   : :>
                                                                   Distribution of housing                     needs j                Concentration’of        housing need by geographic            area or demographic
                                                                                                                                      characteristics     s i’.:‘! i
Magnitude      of community                                                  Extent of economic          distress                  Percent of people at or below the poverty level; per-capita or
deye!opp$.p+,             .,                               ,I ., r   “. / :, ,. iI!       :     :            :    ., : ,! ‘,.      household,,income;,rate    of growth in retail and manufacturing
          1,,   :;,.                                                     ,:,                                                     i employment;‘unemplbyment         rates or underemployment       rates or rate
   : :,       .I;,    ,;_“:                                ’ ! .“/,,. ;.,:,.                 I,                            .”      of long’teirh unemployment;      new capital expenditures    (investment     in
                  ).I>,,,                                               j’,         j -.                           I II            new plant and equipment);      amount of retail sales, service receipts
                             .,                          I             ‘5                                    ,,                    (jn’come’,froni the service sector), or wholesale trade; number and
                                                                                       ,.   I(                                     type.of %bsinesses; crime rates by crime type
                                                                                 Extent of physical    distress                 ’ Extent of’garba.ge4ittered    streets; number and extent of unpaved                  or
                                                                                                                                  broken streets; number and concentration            of condemned    or
                                                                                                                                  abandoned     buildings; percentage     of streetlights    missing or
                                                                                                                                  ineffective; extent of inadequate     drainage and sewage facilities
                                                                                 Distribution   of community                     Concentration  of community development                need by geographic          area
                                                                                 development      needs                          or demoaraphic   characteristics

                                                    /’                                     > Page 60                                               ‘,     GAO/PEMJl-90-9        Partnership     Projects    Framework
                                              Appendix VI
                                              Evaluating Public-Private   Partnerships   From
                                              a Federal Perspective

                                               The specific needs addressed’by federal housing and community devel-
                                               opment projects do not have to be the same as those of the individual
                                               projects they support. Thus, in any given evaluation situation, it would
                                               be necessary to specify the needs that are being addressed by the pro-
                                               gram or programs being evaluated. Such information would be obtained
                                               from a review of the authorizing legislation, program regulations, and

I                                              other program documents. The program could succeed in meeting these
                                               purposes, even if some individual projects did not meet all their own
                                               goals. For example, a particular housing rehabilitation project might use
                                               a partnership model to empower the members of an ethnic minority by
                                              giving them experience in managing an enterprise, but the supporting
                                              federal program might be designed to rehabilitate rental housing for
                                              low-income households, regardless of ethnic identity or the goal of man-
  I.                                          agerial experience. Even if the project failed to meet its empowerment
:- :                                          goals, it might well contribute to the federal program’s success in fulfil-
  f                                           ling its housing purpose.

       How Well Is                              Table VI.3 presents 60 criteria against which federal programs may be
                                                assessed in terms of their efforts to monitor public-private partnerships:
       Impl,qxieqtation Of                     oversight of resource use ‘and administrative oversight. The first deals
       Federally Assisted :                    with how well ,federal program managers are able to oversee the use of
                                               federal funds (or other resources) by partnerships, the second with the
       Partnership Projects
                                               degree of oversight federal agencies maintain over projects to ensure
                                               that they develop evaluation plans, make progress against milestones,
                                               me.et project objectives, and are coordinated with other federal efforts.
                                              ‘while’having,the means of addressing these criteria may be a necessary
                                               condition for sudcessful,‘implementation,   they are of course not suffi-
                                              .,c+nt to ensure success.
       .(                       _.’   ‘_   ,.                       t :
               ,.   .,                            ’            ‘,
                                       t                  II    ‘,


                                             Page 61                                        GAO/PEMD-90-9   Partnership   Projects   Framework

                                                                              Appendix VI
                                                                              Evaluating Public-Private    Partnerships   From
                                                                              a Federal Perspective

        Table Vl.3: Monitoring the Implementation of Federally.Assisted                               Partnerships
        Criterion                        Indicator                                                      Measure
        Oversight    of,resource          use                Accountability      for resources            Reporting requirements    for projects and compliance;  existence            and
                    ..                                                                                    severity of sanctions for noncompliance;   participant awareness             of
                                                      L’             :
                                                             Level of oversight       effort              Personnel assigned to oversight in relation to number of projects;
                                                                                                          frequency and thoroughness    of agency audits of project finances;
                                                                                                          frequency and severity of penalties assessed for noncompliance;
                                                                                                          evidence of waste, fraud, and abuse
        Administrative        oversight                      Review of project        plans and           Extent of prereview of project operational plans, evaluation plans,
                                                             operations’ i                                and’feasibility studies; degree of oversight of the selection of
                                           .’                         ‘,,                 ,.              projects; degree of regulation over project operations; extent of
                                                                                                          monitoring of, progress against plans; internal control; agency review
                                                                                                          of project evaluations
                                                              Coordination     with other federal         Extent of efforts to.avoid   duplication:    success      in leveraainq
                                                                                                                                                                              - .
                                                             iprograms                                    resources
                                                ,.     “,,

                                                                         The oversight of resources involves both the requirements that grantees
                                                                         account for their use of federal funds or other resources and the efforts
                                                          ! ‘,           of federal agencies to enforce those requirements. Thus, the first indica-
                                                        .;              tar  of oversight of resource use is.the degree of accountability to which
                                                                        project managers are held. ‘This can be measured by the repprting
                                                          ‘,, )! requirementsimposed on the grantees, the extent of compliance with
                                                       .”              those requirements .’ and the severity of penalties for noncompliance (as
                                                                        well as particiI3ant aw’arenessof those sanctions). (These measures
                                                               ‘, clearly relate’to the discussion of public accountability in appendix IV.)
                                                                        At ‘the same time, oversight requirements are unlikely to be effective
                                                                        ‘without active efforts ‘by federal $rogram managers to enforce those
                         .,          ,’              _.,        ,I ’ ’ ‘requirements: Thus; the secondindicator of oversight is the agency’s
                                                                    ” level of effo$” which ‘can be measured by the commitment of personnel
                                                                        to this function in relation to the number of projects to be monitored, the
                                                                        frequency and thoroughness of agency audits of project finances, the
                                                                        frequency and severity of penalties actually assessedin cases of non-
                                                                        compliance, and evidence of waste, fraud, and abuse in the program.
                                                                        Data for these measures, and those described in the preceding para-
                                                                        graph, generally should be available through a review of applicable stat-
                                                                        utes and regulations, agency records, and audit reports.

                                                                              However, reaching conclusions on program performance against these
                                                                              criteria may prove complex. While some level of oversight is necessary
                                                                              for ensuring that program resources are directed to the needs and
                                                                              targets intended in legislation, excessive oversight activity could reduce
                                                                              the successof the program. This is because oversight activities generally

                                ,.                   : /‘,                    Page 62                                            GAO/PEMD-90-9        Partnership     Projects   Framework
                                                             Appendix VI
                                                             Evaluating Public-Private   Partnerships   From
                                                             a Federal Perspective

                                                             require that some.resources, including money and staff time, be,diverted

                                                             from program implementation; This can reduce the effectiveness of the
                                                             program both by discouraging potential mediating agents or partnership
                                                             participants from pursuing projects in the first place and by reducing
                                                             the level of resources actually used to meet needs in projects that are

                                                               No standard of optimal oversight is available to resolve this issue. Thus,
                                                               an evaluation must consider not only evidence of insufficient oversight
                                                              *but also the possibility that such efforts have been more than sufficient.
                                                               In part, the latter could be reflected in the patterns of resource use by
                                                            1programs (for example, if a relatively high proportion of funding is used

 i                                                             to cover oversight activities) and by evidence of the unwillingness of
                                                               likely partnership participants to take advantage of the program in com-
-..I    ,,
                                                               parison to programs with less rigorous oversight activities.
                                                              In addition, some of these measures could lead to quite different inter-
                                                               pretations. For example, an evaluator’s finding a great deal of evidence
              _,‘.           I ,.              ”               of waste, fraud; and.abuse in a,given program could reflect poor moni-
                                             ./         I, toring, but it! also might be mareflection of higher-than-normal willing-
                                                           :I:,nessSoreport and, thus, an indication that the agency was being
                                                              especially vigilant. Similarly, little evidence of such problems could indi-
                                                              cate sound managementor a failure to provide thorough oversight.
                                                              Thus;.the evaluator would have to take account of all the,evidence of
                                                              oversight efforts before reaching a firm conclusion on this point:
                                        .:                         ,..!,’                  “’                       *               ,!
                                                              The second oversight criteri0.n srefersto administrative oversight. Under
              : ./              ,,” ”                         this rubric, an evaluation would take account of the extent to which’the
       :,,.          (,,       ,, ‘!’              .’         federalagency monitors the planning, selection, and operation of part-
                                                              nershipprojects, aswell asthe extent to which program efforts are
                                                              coordinated with other programs. Since most projects are developed by
                                                              state and local mediating agents, the role of the federal agency often
                                                              may be to ensure that those agents are adequately reviewing project
                                                              plans and operations and that they are monitoring results. A particular
                                                              area of concern is the extent to which the agency oversees the selection
                                                              of projects to which federal funds or other resources will be directed.
                                                              This relates, in part, to the honesty and integrity of project management
                                                              discussed in table.IV.3.
i                                                            An important element of agency administrative oversight is the prere-
                                                             view of project evaluation plans and review of the results of evalua-
                                                             tions. This is needed to determine whether projects are successfully
                                                             meeting both project and program objectives.

                     .,                                 :    Page 63                                       ‘GAO/PEMD-90-9   Partnership   Projects Framework
                                                                 Appendix VI
                                                                 Evaluating Public-Private      Partnerships   From
                                                                 a Federal Perspective

                                                                 Finally, administrative oversight. also involves ,coordination with other
                                                                 programs, federal, state, local, or private. This coordination should be
                                                                 aimed at minimizing duplication of effort between the program under
                                                                 review and other programs and identifying opportunities to .leverage
                                                                 resources from other sources to achieve maximum results. (These issues
                                                                 were discussed in appendix III.)

                                                                  Of course, some conflict between the avoidance of duplication of effort
                                                                  at the federal level and the leveraging of resources from multiple
   I’               ’                                             sources at the local level is possible. For example, a project that only
            \                                                     usedfunds from a single federal program rather than from several
                         yi                                       would by that fact be allowing the Ifederal government to avoid duplica-
                                                                  tion of effort. However, the project may miss opportunities to leverage
                                                                  local resources if these resources are tied to the unused federal funds.
                                                                  While duplication of effort, in general, is to be avoided, one’of the major
                                                                  reasons for implementing housing and community development projects
                                                                i through public-private partnerships is to take advantage of opportuni-
                                                   ,’             ties to ,combine resources from many sources. Thus, to the extent that
                                                                  the,partnership approach allows for the leveraging of resources that
                                                                  otherwise would not have been available or sufficient to meet a given
                                                                  need, the problem of duplication does not arise.


                                                                 We have identified three criteria against which to measure program suc-
How Stic&sfful Are                                               cess through public-private partnerships: (1) the degree to which the
Feder&ly -.A&stbd                                                intended housing or community development objectives are achieved
 .’  ,,‘,
                  ,:’ ,‘,
                                                                 through partnerships, (2),the success in targeting resources to the
                                                                 intended populations, and (3) the relative success of different public-
         ‘j,                 ‘;:   J ;           :,(             private partnership arrangements; compared to each other and to non-
           ,‘.          .,               ” :           .; ..-    partnership~approaches:($eetable    VI.4.)
             _,i?                                                                          ‘(.’
        ,t,             2’                (:>/            .,.
            ,,                                                                ,        ,,,” /
             ,’8                                                        ,,        .I

                                     1’                          Page 64                                              GAO/PEMD-90-9   Partnership   Projects   Framework
                                                             Appendix VI
                                                             Evaluating Public-Private        Partnerships   From
                                                             a Federal Perspective

Table VI.4 Program Success Through Public-Private Partnerships
Criterion                      Indicator                       Measure
Achievement        of intended                Improvement      in housing       throuoh      Number of new housina units produced or housina units renovated:
objectives                                    partnerships                  -         -      number of households?provided  with rental assistance and amount’
                                                                                             of assistance’
                                              Improvement      in community                  Net job creation; businesses    supported      and amount     of support;     new
                                              development      through                       business starts assisted
Targeting       success                       Extent to which program is                     Proportion of resources directed to target population units (e.g.,
                                              directed to target population          units   families or communities)   or to target populations with special
                                                                                             characteristics (e.g., low income or elderly)
Comparative     success      of public-        Acquisitibn of resources through              Amount of private   resources   obtained      by supported    partnerships;
private p,artnerships                         ,partnerships                                  leveraging ratio
                                          i    Relative success of. partnerships             Extent to which partnerships    meet goals compared to other
            ”                                                   :                            arrangements;   relative success of different partnership

                                                            Table VI.4 illustrates, the application of these criteria. The measures
                                                            listed first in the table, regarding achievement of intended objectives,
                                                            illustrate the general purposes of, housing and community development
                                                            programs. In aqparticular          evaluation, the goals of the program being
                                                            reviewed would have to be.specified more precisely. However, in general
                                                            terms, ,evaluations of housing programs would focus on the number of
                                                            housing units produced or renovated or on the number of households
                                                           provided with rental assistance. For community development programs,
                                                           the focus would be on such measures as net job creation, number of busi-
                                                           nesses supported, and new business starts. (See table V.l for further
                                  .,                       measures.)                            .
                                ,,,                          ‘,/
                “8,.      I,, ” !                          A second
                                                      ., I’_..   ‘!I,‘  criterion
                                                                               ; .,: of success in’attaining program goals concerns the
                                                           extent to which programs are ,able ,to reach target populations. Most
                                                           housing and community development programs are targeted to individu-
                                                           als and families or to communities, but some target small business, prop-
                                                           erty owners, or other units. Within these categories, programs may be
                                                           targeted to units with specific characteristics, such as low-income fami-
                                                           lies, the elderly, or the unemployed. Part of the relative success of a
                                                           program depends on the extent to which benefits flow to the population
                                                           for which it is intended. To some extent, partnership approaches may
                                                           complicate efforts to target these groups by bringing to bear the goals of
                                                           the private sector partners as part of project design.

                                                             The third criterion of success involves the evaluation of different part-
                                                             nership types compared to each other and to nonpartnership
                                                             approaches. That is, the evaluation would take account of the extent to

                                                             Page 66                                                GAO/PEMD-90-9       Partnership   Projects   Framework
                                                                                                                  Appendix VI
                                                                                                                  Evaluating Public-Private   Partnerships         From
                                                                                                                  a Federal Perspective

         ,,’    ,;                             :-
                                                                                                              which the use of public-private partnerships in administering the pro-
                                                     ‘,                                                       gram resulted in more or less successthan might have been achieved
                                                                                                           * without this approach (for example, by working through public-sector
                                                                                                              channels only). Ideally, this could be done where a particular program
                                                    i’.           ‘:                                        ’ was use,d,to’fund a large number of othervirise similar projects, some
                                                                                                              with the partnership feature and some without.

                                                                                                                 In practice, it is unlikely that any given program would include large
                                                                                                                 numbers of comparable partnership and nonpartnership projects. Some
                                                                                                                 programs primarily fund partnership projects, while other programs
                                                                            ., ,,,                               generally. do,not. Moreover, partnership and nonpartnership cases are
                                                                                                                 unlikely to rbecomparable in all other relevant dimensions. ,Finally, even
                             /1                                                                                  in the event that a set of cases tiere identified, it is unlikely that compa-
                           ,: .
                                                                                                                 rable .data~wouldbe available for them. Nevertheless, if such conditions
                                                                                                                 were obtained, the typeof analysis described above would be best.
                                        :;                                            5                             At a minimum, however, an evaluation could take account of the extent
                                                           . ..(                                                    to,which the use of public-p&&partnerships          resulted in the acquisi-
                                                                                                                 ” tion of ~additi~ohdl,res~~rce~    to support program efforts. Both the total
               ,!,               ~..            ,, :-I’.‘,
                                                       ,                                                          ‘am&u&of original~resources acquired and the leveraging ratio (that is,
                                                   . ..’                                                         ’ the’ ratio ‘of Ijrivate.funds to the’pdblic investment) would have to be
           _1 /,.                         : ,/I : ‘.,.,                                                             considered. ‘&&en thatpartnershi& are recommended as a way of
                                                                                                                   ‘imI&ing s&port for ij’rogr&s by tapping into the private sector, suc-
                                                                                                                    c&s on’
                                                                                                                         ., these measures      is crucial.
               z                                                                                                                       I      ‘I I,        :
.,,,:’                                                                      <,,. .j.. .
                                                              If the evamation centered on ‘a ij’articular federal program, a sample of
                                                              partnership projects assisted by that program could be studied. How-
  * 1,                             / .,,,                     ever, if the evaluation were concerned with evaluating the partnership
                                                      ‘.,i mechahism’across~l.&ograms;the. study sample would have to be
                                   .‘,‘I ,,,,?‘,             ‘b&d&kj ,. /.,, ’ I, ,‘,.I’.?,’‘i $I;
                           :       .’              .>.A
                                                    ’          ,., ‘:,> I’ ,           _/ .       ,..,
                           , ,’ :, ,:.        ./    ,:,             !    /:          ,t~
                                            : .,     >,., _ ‘, :        I’ ,,      ‘!._‘.,           I_
                                      i(‘,...,.  i                            ‘,        !’ 51.
                              :                                                  :

                                                                            8.                  _’
                                                     ,,.           .,        ,.                                        1
                            I/                                                    :                  ( ,                                        I’        :,   ,      5.
                                        ‘..                       ;;                                        ,’
                 -i               .,.                                  ‘!                                                                            .:

.,/ .,                ?,    ~,                I.           ,*.:                           .‘,               ‘.
                                                                                                                  Page 66                                                  GAO/PEMD-90-9   Partnership   Projects   Framework
Appendix VII

‘k-pert Reviewers

                       . ...,                               The following experts ‘participated. in a meeting to discuss the evalua-
                                                            tion framework for. public-private partnerships. They also reviewed an
                                                            initial and a final draft of the framework.

                                                            Scott Fosler, Vice President and Director of Government Studies
                                                            Committee for Economic Development
                                                            Washington, DC.

                                       ,                    Arthur T. Himmelman, Director
                                                            Public/Private Initiative Project
                                                            Hubert H. Humphrey,Institute of Public Affairs
                                                  ,         University of Minnesota,
                                                            Minneapolis, Minnesota

                                                            Barbara Lipman, Project Director
                                                            Advisory Commission on Intergovernmental Relations
                                                            Washington, DC.

                                                            Harold Seidman, Guest Scholar
                                                            Johns Hopkins Centerfor the Study of American Government

                                  ..                        Alice Shabecoff, Executive Director1
                                                      ’     Community Information Exchange
                                                            Washington, D.C.
                      ,::                   ‘.                 Diane Suchman, Director
                            .,                                 Housing and Development: Research
               :; :              .’ ., .,                     ‘Urban ‘Land fnstitute
                      :                                   ,.,,,,Washington,
                                                                    : ., DC. .’
                                             <:              :

                                             :              James Vitarello, President
                                                            James Vitarello Development Associates, Inc.
                                                            Washington; DC.

                                                            ‘Alice Shabecoff did not participate        in the panel meeting but did review drafts of the evaluation

        ‘/                                                  Page 67                                                 GAO/PEMD-90-9      Partnershid   Projects   Framework
Appendix VIII

Agency Comments

                                                                                                    U.S. DEPARTMENT     OF HOUSING    AND URBAN     DEVELOPMENT
                                                                                                                      WASHINGTON.  D.C. 20410-7wO
               .,.   ,“.I

                                                                                      JAN 26 K%fl

                                                                                     MS. Lois-ellin     Datta
                                                                                     Director     for Program Evaluation
                                                                                        in Human Services
                                                                                     Program Evaluation       and Methodology
                                                            I., (.'       '             Diy,ision
                                                                                    'General Accounting Office              '
                                                                                    Washington,      D.C. 20548
                                                                                     Dear Ms. Datta:
                                                                                            ,',., :
                                                              :I.                            Thank you for the! opportunity        to review GAO's draft
                                                                          :!        'report'entitl'ed,      DPartnership'Projects:        A Framework for
                                                                                     Evaluation       of PublicTPrivate,   Housing and Development
                                                                                            I want to compliment you ,and your staff        in the Program
                                                                                    Evaluation    and Methodology Division       for developing   an
                                                                                    excellent    report and, also 'for'their     initial  fact sheet
                                                                                    entitled,    "Partnership   Projects     - Federal Support for Public-
                                                                                    Private    Housing and' Development 'Efforts."
                                                                                         This solid,      well-,written-,report         provides a comprehensive
                                                                                  set of consideration's         and measures for conducting              evaluations
                                                                                'of pub'lic-private        pa,rtnerships:       id housing and community
                                                                                 development.         Nevertheless,,as          the authors of the report
                                                                                  indicate,      such evaluations         are difficult       as a result     of the
                                                                                  lack of readily        avai~lable;,,reliable         data and the high costs
                                                                                 associated       with collecting         the needed data.         Furthermore,
                                                                                 even with the report,           it is difficult         to apply selectively
                                                                                 the framework...in ev,aluating:.theV,,specific               programs, projects,
                                                                            ,., and problems         associated
                                                                                            ,.I'. . ', "i I s: .". ; with    public-private       partnerships.
                                                            .Ii         : ,: ,..'i';' :"
                                                                                         It would be bur,densome,and,>costly               for Federal agencies
                                                                                 to conduct comprehensive evaluations                   of public-private
                                                                                 partnerships        using the total         framework as described in\the
                                                                                 report.        There would need to be substantial                further
                                                                                 discussion       of the precise requirements              for evaluations      before
                                                                                 the Department could endorse such efforts.

    ,, ,   ’                    ;I,            .
                                                       ,'                     ',;

                            ’         i,           I              ‘,-
                                                                                Page 68                                               GAO/PEMDBOB           Partnership   Projects   Framework
         Appendix VIII
         Agency Comments

               The information      in'the   report may be helpful   to some
         local and state governments .who wish to undertake evaluations
         of public-private      partnerships     in order to improve monitoring
         or to provide better technical          assistance.  To encourage such
         evaluations     by state and local governments,      there are several
         reasonable steps that could be explored by GAO:
                             (1   In the long run, it may be beneficial          to
                                  determine first    specific   evaluation     questions
                                  that would be,of most value to state and local
                                  governments:    what are their priority        needs in
                                  understanding   the public-private       partnership
                                  process and the outcomes of that process7
                             (2   Based on such a determination,          it would be
                                  worthwhile     if the Program Evaluation       and
                                  Methodology Division       of GAO would undertake
                                  one or,. more prototypical      evaluations    of
                                  public-private     partnerships     for housing and
                                  community development efforts,          in large part
                                  to assess the availablity        and quality    of data
                                  and of the cost of undertaking          such
                             (3 ) Finally,    it would be very useful if GAO would
                                  prepare a guide book that can assist       those
                                  entities    wishing to conduct similar
                                  evaluations     of specific public-private
                                  partnership     endeavors.
                    In closing,     I want to acknowledge my appreciation                 for
         your       inviting    participation  in this effort.
                    I look   forward    to continued     cooperation      in these

                                                 Anna Kondratas
                                                 Assistant  Secretary

          Page 69                                         GAO/PEMD-90-9     Partnership    Projects   Framework
Appendix IX

Major Contributors to This Report                                                   :                ”         ’

Program Evaluatick     Patrick Grasso, Assistant Director
                       Susan Labin, Project Manager
and Methodology        Mary L. Westcott, Project Manager
Division               Leslie J. C. Riggin, Social Science Analyst
                       Robert L. -York, Assistant Director

                      Page 70                             GAO/PEMD-90-9   Partnership   Projects   Framework
                                                                 Abt Associates. The Urban Development Action ?&r-it Program: ACorni: ’
                                                                 prehensive Evaluation Design..Washington, DC.: August 25, 1981.
                                                                                                                   .’     .
                                                                 Adams, Salisbury. “Public Private Sector Relations.” The Bureaucrat,; , ?
                                                                 spring 1983, pp. 7-10. ,.

                                                                 Ahlbrandt, Roger S., Jr. “Public-Private Partnerships for Neighborhood
                                                                 Renewal.” Annals of the AAPSS, 488 (November 1986), 120-34.

                                                                 Ahlbrandt, Roger S., Jr., and Clyde Weaver. “Public/Private Institutions
                                                                 and Advanced Technology Development in Southwestern Penn-
                                                                 Sylvania.” APA Journal, 53 (autumn 1987), 40-77.

                                                                 Angotti, T. “Housing Strategies: The Limits of Local Actions.” Journal of
                                                                 Housing, September-October 1986, pp. 197-206.

                                                                 Arndt,‘R. “Task Force Gives Findings on Public-Private Partnerships.“0
                                                                 Nation’s Cities Weekly, 5:50 (December 30,1982), n.p.
                                                                     :               ._         *,
                                                                 Bachelor, Lynn. “Evaluating the Implementation of Economic Develop-
                                                                 ment ,Policy: Lesson’s from Detroit’s Central Industrial Park Project.”
                                                  ‘,             Policy Studies Review, 4:4 (May,1985), 601-12.

                                                                   Bailis, Lawrence Neil. “Private Sector Initiatives, Vegematic of the
                                                                  ,198O’s.” New England Journal of, Human Services, Spring 1984, pp. 1l-
                                                                   18.             .:
                                          .,                        ”          ,j.
                                                                   Bendick, Marc J., and Phyllis M. Levinson. “Private Sector Initiatives or
   .I _:.... ‘, ., : ‘.                                          ,,Public/Private,Partnerships,l) pp..455-89. In Lester Salamon and M. S.
” ‘,, i: ,’                              ,,                   ,G iLund (eds.). ,The{ReaganPresidency and,the Governing of America.
                                     :                   ,’   ,’ Washington; D.C;: Urban Institute, 1985.
       :       8’          ,.
                                                                 Bernstein, Nancy K., Stephanie Gottile, and Robert K. Yin. Evaluation of
            ,. ..,.                                              the Funding for .Neighborhood Initiatives: First Year Report. Vol. 2. Case
                                                                 Vignettes. Washington, D.C.‘:COSMOS,July 1986.

                                                                 Black, J. Thomas, et al. UDAG Partnerships: Nine Case Studies. Wash-
                                                                 ington, D.C.: Urban Land Institute, 1980.

                                                                Bradford, Calvin. “Private Sector Initiatives and Public Center Account-
   .                            //                              .ability.” APA Journal, summer 1983, pp. 326-35.

                      ,,                      ~   /:;,           Page 71                             GAO/PEMD-90-9   Partnership   Projects   Framework

                                           Bratt, Rachel G., Janet M. Byrd, and Robert M. Hollister. The Private
                                           Sector and Neighborhood Preservation. Cambridge, Mass.: Neighborhood
                                           Policy Research, January 1983.

                                           Brettler-Berenyi, Eileen. “Public and Private Sector Interaction Patterns
                                           in the Delivery of Local Public Services.” Government Finance, 9:l
                                           (March 1980), 3-9.

                                           Cavanagh, R. “A Businessman’s Perspective.” The Bureaucrat, spring
                                           1983, pp. 21-22.

                                           .Celis; W., III.. “Public-Private $153cMillion Program to Build Housing to
                                           Be Disclosed Today.” Wall Street Journal, October 11, 1988, p. 22.

                                           ,Clarke, S. E., and M. J. Rich. “Partnerships for Economic Development:
                                            The UDAG Experience.” Community Action, 1:4 (1982), 51-56.

                                           ‘Clay, Phillip L. Neighborhood:Partnerships in Action. Washington, D.C.:
                                           Neighborhood Reinvestment Corporation, August 1981.
                            :                  Community Information Exchange,.Neighborhood Development Demon-
                                               stration Program, Case Study Examples in Neighborhood Development
                                               and Grass Roots Fundraising. Washington, D.C.: November 1987.
                                               Council for Northeast ,EconomicAction. Local Economic Development:
                                               Public Leveraging of Private Capital. Washington, DC.: U.S. Department
                                               of Housing and Urban Development, April 1980.
                       ,”                                              8:,, 1 ,_
                  ,I   .’       ,,

                                               Council for Northeast Economic Action. How Banks Participate in Local
I.,’              ?,.,          .,;,.    :. :I Economic:Development: Five.Models. Washington, DC.: U.S. Department
                                        : of’Housing and UrbanDevelopment, January 1982.
,’               .’                        ‘Council of State Community’Affairs Agencies. States and Economic
             ’                              Development: Expanding Oppo’rtunities for Employment and Self-Suffi-
                                           -ciency. Cleveland, Ohio: June 21-23; 1988.
                                           Coyle, J. “Local Government Perspective.” The Bureaucrat, spring 1983,
                                           p. 17.

                                           Cullinan, Terrence. In Good,Company: Corporate Community Action and
                                           How to Do It in-Your Community; Menlo Park, Calif.: SRI International,
                                           October 1982.

                                           Page 72                            GAO/PEMD-SO-9   Partnership   Projects   Framework

                                                                                                                                                  Davis, P. (ed.). Public-Private Partnerships: Improving Urban Life. New
                                                                                                                                                  York: Academy of Political Science, 1986.
                                                                                                                                                  DeSeve,Edward G. “Financing Urban Development: The Joint Efforts of
                      I,,~                       /!.’
                                                                                                                                                  .Governments and the Private’ Sector.” The Annals of the American
                                                   .                                 .,                      ,,                                   Academy, 488 (November 1986), 68-76.

                                                                Downs, Anthony. Neighborhoods and Urban Development. Washington,
                                                                D.C;: Brookings’Institution, 1981.
                                                          I                             .,,        .’
                                                               Dreier, P. “Community Based Housing: A Progressive Approach to a
                          !‘.                  ,: I:           New Federal Policy.” Social Policy, fall 1987, pp. 18-22.
                    . ..         :                      , ,“;:              ., .       !
                                                               Duckworth, Robert P., John M. Simmons, and Robert H. McNulty. “Part-
                                                               ners for Livable Places.” The Entrepreneurial American City. Washing-
                  ;,,             ‘,         .              ;I ,ton’,D.C.: USDepartment of Housing and Urban Development, 1986.
                                      ,, .’ .’
                                                                “Election 8&:.Eight .&utdidates Speak.” Journal of Housing, January-
                                                               February 1988, pp. 23-27.
                      ,.i ”                      ./ ,.             ,’        .‘<     ,’ I(,.’         ,
                            ! ,I ,‘,’            ,/            +Erdmann, Robin J.ibHarlan Gradin,       and Robert 0. Zdenek. Community
                                                               Development Corporation Profile Book. Washington, DC.: National Con-
                                      5 _‘~.         /,‘$ #‘,: mgress for Community Economic Development, September 1985.
                                                                             ; 1.’I. ;“.*         ‘,
                                                               Ervin, 0. L. “The State-Local Partnership and National Objectives: An
                     ‘, ;’               ,’          :         Examinationof the .qDBGSmallCities Program.” Policy Studies Journal,
                                                                13:3 (March’X985)j 634-42.: ;
          ‘i,;       /‘:                    .‘, .r i(        ,,‘,           ;               “.‘,L,:
                                                                                                     : ,,Fastrtip, Jerry 6. ‘Why .Doesthe Money Go Where It Goes?” GAOJour-
                                                                                                             ,’              ::’
     :: :.’7’(I(/,                                                    ; ,j ;~,,,
                                                                           .,I :,;‘.,, ,‘,:;.;‘~.,;(y’j
                                                                                                    ..I..@; 6 ~sur~e~;~989j),;~3-59,, 1’;: :f.,,i*
                  ,!!        :',     ;'.;
                                                               l'f'                 Fletcherj’rI’l VW.,andi,J.M. McGwire; +‘Government and Private Business:
                                                                                     ,,:.             ;:;d              1      .,            ,'

                                                             ,!;.:*<.I ,, i,.~ ‘(., New Relatiohships~Emergin~~”National Civic Review, October 1980, pp.
                 :                                    ,_’                :‘.,. i.,      ;, ‘. 1 , s,,:” 18, . .
                                                   ‘:     ,. :                       Fosler;RScott, and Renee A. Berg&. Public-Private Partnerships in
r            ,:                                 ‘, ,, ,,,; :     : f                -American Cities. ,Lexington, Mass.:’Lexington Books, 1982.

                                                                                                                                                   Fox, Siegrun F. “Who Opposes Public/Private Financial Partnerships for
                 .:                           II.          I:                                                                                     : Urban Renewal? A Case’Study.“~Journal of Urban Affairs, 7:l (n.d.), 27-
                             ,;               “.,‘.
                                                ;, : ., ;, ‘I,,.,
                                                              :                                                                     i              ,440.    ;:    ,;i               J

    I..              ,             ;/,.:           I, ,,”             .‘.        C,“,                  “(         ,,, “.,,              ,‘,,      ,page,g                            GAO/PEMD-90-9   Partnership   Projects   Framework
     I                                                                                                                                                                                            ,


                                                                                                                 Frieden, Bernard J.,’ and Arthur P. Solomon. The Nation’s Housing: 1975
                                                                                                                 to 1985. Cambridge, Mass.: Joint Center for Urban Studies of the Massa-
                                                                                                                 chusetts Institute of Technology and Harvard University, April 1977.

                                                                                                                 Garn, H. A., N. L. Tevis, and C. E. Snead. Evaluating Community Devel-
                                                                                                                 opment Corporations-A Summary Report. Washington, D.C.: Urban
                                                                                                                 Institute, March 1976.

                                                                                                                 Glickman, Norman J. (ed.). The Urban Impacts of Federal Policies. Balti-
                                                                                                                 more: John Hopkins University Press, 1980.
                                                      i                    ,I
                                                                                                                 Gold, Kenneth A. “Managing for Su&ess: A Comparison of the Private
                                                                                                                 and Public Sectors.” Public Administration Review, November-December
                                                                                                                 1982, pp. 56875.          ,

-.                                                                                                               Goedert, Jeanne E., and’ J. L. Blake. Innovative Projects for Neighbor-
                                                                                                                 hood Revitalization: A Case Study Evaluation. Washington, DC.: Urban
                                                                                ,I                               Institute, December 1.980.

                                                                                                                 Goedert, Jeanne E., and J. L. Goodman, Jr. Indicators of the Quality of
                               (”                                 .                                              U.S. Housing. Washington, DC.: Urban Institute, September 1977.
                                                                                           :.I ,’                      ..1      ,; .’ ,>
                                                                                                                 Grindley, iyilliam C. Making and Measuring Corporate Urban Ventures.
                                                                                                                 Menlo Park, Calif.: SRI International, June 1980.
                                ,.                    ,,     .’
                     ‘.         .;’                    ‘, ,.                                                     Hanson, Ranae,:‘andJohn McNamara. Partners. Minneapolis: Dayton-
                                                                                                                 Hudson Foundation; :I981..  I
         ,.:;,                  )‘.                    ,,             .,                   ,,   ‘,    __   ..Hartman; Chester. AmericaTsHousing Crisis: What Is to Be Done? Bos-
                                                                                                            ton, Mass.: 1nstitute:for. Policy Studies, Routledge and Kegan Paul, 1983.
                                              :              ,I
                                                                                 ‘/                          ,Hentzel,,S.W., et al;.E%ploring.Urban Action Options for Local Firms.
                     .: :                                                                                    Menlo Park, Calif.:. SRI,International, October 1980.

                                                                                                                 Himmelman, A. T. “Working Collaboratively for a Change: Community
                                                                                                                 Problem Solving for the 1990’s.” Public/Private Initiative Project, H. H.
                                                                                                                 Humphrey Institute of Public Affairs, University of Minnesota, Minne-
                                                                                                                 apolis, n.d.
                          ‘;                      <
                 :                                                                                   ,.    ‘..

                                    ,’                ,,‘,
                                                                                      ‘.                     Howland, Marie. Urban Development Action Grants: Five Case Studies.
                                                                                                             College Park, Md.: The Institute for Urban Studies, University of Mary-
                                                                                                             land, nd.

           il        ;,                                                                                     Page 74                                G+O/PEMD-99-9   Partnership   Projects   Framework
                                                                       Howland, Marie, and Ted Miller. Job Creation in the Rural UDAG Pro-
                                                                       gram. College Park,.Md., and Washington, DC.: University of Maryland
                                                                       and Urban Institute, May’1988

                                                                       Howland, Marie, and Ted Miller. Urban Development Action Grants to
                                                                       Rural Communities. College Park, Md., and Washington, D.C.: University
                                                                       of Maryland and UrbanInstitute, May 1989.
    ;                                                  .’                   International City Managers Association, Management Information Ser-
                                                                            vice Report. Developing Public/Private Approaches to Community Prob-
                                                                            lem Solving, 14:7 (July 1982), 1-18.
                              i                         .,        ,,, .’                            .‘,
                                                                      . Jacobs, SS., and%. A. Roistacher. “The Urban Impacts of HUD'S Urban
                                                                             Development Action’ Grant Program, or, Where’s the Action in Action
                                                                             Grants?” In N. J. Glickman (ed.). The Urban Impacts of Federal Policies.
                              ”                                              Baltimore:        Johns Hopkins University Press, 1980.
                                                                           .‘I ,’ ::                    _          ,,
                                                                             Jamison, David C., and Alan R. Siegel. “Federal Role in Local Partner-
                                             .’                             ships.‘? The Bureaucrat, spring 1983, pp. 27-29.
I                   ‘,’           ‘,         ,‘I.             ,,(             -1,          ,,,,    :a. ,:,.I-     .., ”
                                                                            Johnson,fV;, and T. Kolderie;.,‘fPublic/Private Partnerships: Useful but
                                                                            Sterile.” Found:ation News, March-April 1984, pp. 29-33.
        : ,‘,                                      ’             ,*I...I:.” ,,:: ,’,., i,.
                                                                                       ‘, ‘ , ,’ ,” ‘. ; ,‘..I‘_‘1 ;’ ”
                                                                            Kaplan, Marshall~ and James.Franklin. HP #3, Federal Housing Agencies
                                                                            and National Housing Policies, Institutional Roles, Relevance and
                      ”           ’               .’ ,’ ,.,    ”\,.. ., .-ResponsibilitiesiesCambridge,Mass:i’MIT Center for Real Estate Develop-
                                                                      men& April   ‘1988,      IL>,*,:   ,: :‘.: 1

        ,...        .,’
     ., ,,I          ,, .,.. ,’ ;, ., ,I    ; ” ;Kettl; Donald:~F~.Governmentby, Proxy: Mismanaging Federal Programs.
    ,/ J:, i,,:,, 5:” ;, ;‘,,;: ,,. .‘,/“,,‘I ‘. _’Washington; D:~:.::,:ongres~ion’al’Quarterly, n.d.
                                                      ._ .,                    ..
                                                                -,; ’ ,.II i.,‘):
                                                                            .?“,< 1 ,, \; ! :’: ,
                                                   ~Langston,Stuart. “Public-Private Partnerships: Hope or Hoax?”
                                         /’ ~ ,’ National..Ciaic,‘!Review, May: 1983;;pp. 257-61.
                  ,.        a., ‘,‘I               ;,,,,; : ,. j_.           .,‘,.  ) .,,/I :
                                                   -Lanier; R., A. Mass&r, and C. Oman. Public and Private Sector Partner-
                                                    ships ,in Housing: A Background Paper. Washington, DC.: Technical Sup- ,
                                  .’              “.port Services, Inc., February.1986.
                                                    Lipman, Barbara J, Post Partnership Analysis: The Feasibility of Evalu-
                                                : ating~.Public/Private~Real’Estate Developments Against the Original
                                                   -Objectives of the Partners; Washington, D.C.: Urban Institute, July 1988.

          ;    ,,                      .I’                      II
                                                                      “Page 75                                  GAO/PEMD-90-9   Partnership   Projects   Framework


                                                                                                                                     Mahmood, Syyed T., and Amit K. Ghosh (eds.). Handbook for Commu-
                                                                                                                                     nity .Economic Development. Los Angeles, Calif.: Community Research
                                                                                                                                     Group, 1979.
                                                                                                                                     Mayer, Neil S. Neighborhood Organizations and Community Develop-
                                                    ’/                                                                               ment. Washington; D.C.: Urban Institute, 1984.

                                                                                                                                  Mayer, Neil S. The Role of Nonprofits in Reviewed Federal Housing
                                                                                                                                 .Efforts. Cambridge, Mass.m:,MITCenter for Real Estate Development,
  ‘I                                                                      ./ !‘,                                                 !1988.
                                                                                                                                                I.   ,,
                                                                                                                                  Merget, Astrid E. Review of Public-Private Partnerships: New Opportu-
                    I ,..I        !:          ‘,.                                                                                 nities for MeetingSocial,Needs; In,Public Administration Review, 46:4
                                                                                                                                  (July-Augus0986), 364-65.        ‘:
       Cl               ‘/

                                                                                                            j                    Miller; ~DelbertC.’Handbook of Research Design and Social Measure-
                                                                                                                                 ment, 2nd ed. New York: David McKav Co.. 1970.
                                                                                                                                 I                                       Y     I
                                                                                 ,,.)‘\.’                                   ,,               :.i

                                                    i Mitchell& C.“‘$heConcept and Use of Social Networks,” pp. l-60. D. C.
                                                                         ! :I’

                                                        Mitchell (ed.). Social Networks in Urban Situations. Manchester, Eng.:
  ,!,,i)              ,‘I’ !, ;. :;A; ,, ( ) ,Manchester.University Press;)19691
                             ., ,‘. ; ‘; I ,,.,; ; I; i.,-, ,‘*.            >:’ ‘,        :,., 1 ; .,
                                                        Mogulof, Melvin ‘B. Citizen Participation: The Local Perspective. Wash-
: j:‘,,<.;.;:    “I ,z ., ,I, .’ ,, n_     -,    :i ‘.ington, DC:.: UrbanInstitute,’ 1970.
             l.L,,dJ ia ?~_(    ,,P, :.,. .I *. If_<,,,_(/I,,$.I ,,,I,.j~.~i,~:$j,<,:.: /// ‘.L      I ,ld
 /;s,.,.,,, .,’I         (’ /,’ ., ,1,I: ‘. _a : Morris; Peter; and MartinRein Dilemmas of Social Reform. Chicago:
                                                       Aldine, 1973.               a: ,-II      I > ij
            y;.:::;j.,>-;         “!>:,‘,*;
                                                    :; ,,,, 1.1:
                                                             .I_(        .,... Jf,C.;,R;‘B: Katz; and L. M. Salamon. Government Spending
                                                                    :.:,, ,/,
                                                                                              :,,I’.,                 ,‘,

                                                     1’1,‘~andtheNonprofitSedtor~in~Pittsbdrgh/Allegheny County. Washington,
                                                           D.C.: Urban Institute, 1985.
                ;: i :!I‘ . I : ‘,,j,,:“;:
                                    s.< :i”,I ..!(,,;.aI/.*                      :,:: ,:,;,,i. ,“_ or, :;.,
                                                        , : 1; 4;., i:;‘~~:~‘i..“,‘y~~
                                                           NationalCenter for,,Housing .Management. Keys to Successful Public-Pri-
                                                                                       (I             i.-   ;
                                                                                             ,:                 ..’

                                                           vate Partnerships for Housing Development. Washington, DC.: 1985.
    , ,:..._
           (, ,:,              ,,, ii .!_,,. ,; . ..^ , “L. ; : ‘(‘T..,                              “,    i
   . i_, i                 ::                 .,
                                             :,. ::,       National,Congress for Community Economic Development. Against All
                                                           Odds: Achievements of Community-Based Development Organizations.
                                                           Washington DC.: March 1989.
             !, ,‘. .,                        i:,,,‘..                            ,, *-‘,                                     NationalCorporation for Housing Partnerships. 1988 Report to the Pres-
            ,,.’                                                   .,’                                                      ’ ident ofthe United States. Washington, DC., 1989..

                                                                    ,:                      ‘,,?                                 Page 76                             GAO/PEMD-90-9   Partnership   Projects   Framework

                                         National Housing Task Force. A Decent Place to Live. Washington, DC.:
                                         March 1988.

                                     National Institute for Advanced Studies. Public-Private Strategies for
                                     Involving Small Business in Community Economic Revitalization: A Pol-
                                     icy Oriented Analysis. Washington, D.C.: July 24, 1978.
                                     Neighborhood Revitalization Project for the Center for Community
                                     Change. Bright Promises,.Questionable Results. N-p.: n.d.
                                     Nenno, Mary K. New Money and New Methods: A Catalog of State and
                                    Local Initiativesin Housing and Community Development. Washington,
                                    ,D.CI:National .ssociation of Housing and Redevelopment Officials, Sep-
                                    tember 1986.

                                    Nenno, Mary K;, and,George Colyer. New Money and New Methods: A
                                    Catalog of State.and Local Initiatives in Housing and Community Devel-
                                    opment. Washington,, DC.: National Association of Housing and Redevel-
                                    opment Officials, 1988.
                                               j        .,.            2
                                    Orr,’ J. Martin. “Building, Public-Private Cooperation in Kansas City.”
                                    Economic Development Commentary, 3 (April 1979), 12-15.
                                    ”“Partnership Models.” National Civic Review, nd., p. 34.

                              ,’    Pattdn.,.MichisielQ. Qualitative Evaluation Methods. Beverly Hills, Calif.:
                                    Sage, 1980.
          /                   ,’           I ,.         ,                                     .I’_”        ;
                                                            >’   ,*

                                   .’Pattoh, ,Michael Q; Utilization Focused Evaluation. Beverly Hills, Calif:
                                    Sage, 1986.
                                                                      ,I        I(
                    :               ,i     ‘:t,                            .:        I   ’        1)   ,   .,.’

                                      Peterson; Pat&; ‘and Kenneth Ku:Wong. “Comparing Federal Education,
                                      and Housing Programs: Toward a Differentiated Theory of Federalism.”
,‘,   ”       .,.       /I.        ‘: <Presented.at the annuralmeeting of the American Political Science Asso-
                                      ciation;~Denvea,Colofado,September l-4, 1982.

                                    Pozdena, Randall J. The Modern Economics of Housing. New York: Quo-
                                   .rumBooks, 1988.       ‘. ;.

                                    President’s Task Force on Private Sector Initiatives, Investing in
                                    America: Initiatives for Community and Economic Development. Wash-
                                    ington, D.C.: nd.

                                    Page 77                                                  ,’                GAO/PEMD-90-9   Partnership   Projects   Framework

                                                                          :       ’                                                          President’s Task Force on Private Sector Initiatives. Building Partner-
                                                                                                                                             ships. Washington, DC.: ,December2,1981-December 8, 1982.

                                                                                                                                            :‘Pressman, Jeffrey L., and Aaron B. Wildavsky. Implementation. Berke-
     .’                                                                       ”
                                                                                                                                              ley: University of California, ,1973.
                                                                                                                                                            i       I

                                                                                                                                             Public and’private Sector Partnerships in Housing and Urban Develop-
                                                                                                                                             ment. Proceedings of the 10th Conference of Housing and Urban Devel-
                                                                                                                                             opment in Sub-Saharan Africa. Sponsored by US. Agency for
                                                                                                                                             Internati,onal Development. Harare, Zimbabwe: February 26-28, 1986.

                                                                                                                                          ‘t : .Real”Estate Research Corpor.ation. Lessons for Cities and States: A
                                                                                                                                              ‘Handbook forkAnalyses of .the Impacts of New Developments on Older
                                                                                                                                               Commercial Areas. Washington,.DC.: spring 1982.

                                                                      -                                                                     Robbins,.Carol T. Private SectorsPartners for Expanding Economic
                                                                 ‘, ,‘...’
                                                                       ,’                              b.,,                                 Opportunities. ,Washington, D.&Council of State Community Affairs
                                                                     .,‘. ’                                            .:                   Agencies, December 1988., :
                                                                                                                                                            ‘.             ./,,
1                                                                                                                                           Rosenthal, Donald B. “Forms.of Local Participation in a Neighborhood-
                                                                ,,_.,                       ,j                  .“’                    ‘I,, BasedaFederalProgram.” Policy Studies Review, 3:2 (February 1984),
                                                                                                                  ‘,                 I. : 279~@j.:: :      ;. j.      ,;.

                                               _,.,                                               :’
                                                                                                                .:,    .              I
                                                                                                                                             Salamon; LesterM., and’Michae1 S. Lund. “The Tools Perspective on
                                                                                                                                             Government Action: Purpose, Promise and Tasks.” Discussion Paper,
                         ;                                  :                     .,                            ;,‘I,,. ,,:‘/,               Urban Institute; Washington;~D.C:,1984.

                         ‘.                             1       ,,                     .‘                                  .Schwartz, D. C., R. C. Ferlauto, and D. N. Hoffman, A New Housing Pol-
                                                                :;:.                        I_”        :: ‘,!r; ,-I,
                                                                                                                  .) I. l -icy~~Philadelphia:‘;Temple~University                Press, 1988.
                                                                                                                                  “” .           ^
                                                                                                                                                                 :,,    :::‘,i”
                                  .        .
                                                                                                                           ,Seidman, H. “The Quasi-World of the Federal Government.” The Brook-
      ,,.-,;, ,( ‘; ’ ;,.:                                                                   I. ,.y,.,;,‘,i.:    ,, ‘.ings iReview) summer (1988; :pp...23&27.
                 !.,.,,                                                                                :‘-,,.
                                                                                                         i          .,‘,‘,.-1L‘, .I.. +>.,,-:,,_
                                                                                                                                                 ‘.         : ,‘;c$ ‘. /_.( :,:
               ,I                                 ./,

                                                                                                                   .I :, Shabecoff, &lice; ‘Neighborhood Housing: Development Innovations,

                                                                                                           .,I,           ,,July-August 1987, pp:~~lO@lO:             ‘( : _..

                                                                                                                                      ‘. -Simmons;J. M::,W. D. Kelleher, and R. P. Duckworth. Business and the
                                                                                                                                          Entrepreneurial American City;. N.;p.:U.S. Chamber of Commerce, n.d.

                                                                                                                                            Simons, L. B..“Overview: Housing Options for the 1990’s.” Yale Law and
                                                                                                                      ,)                    Policy Review; 6:2 (1988), 259-86.
                                                                                                                                                                         ,i ‘,:

          ‘.                 :        :/                             .,                                ,:.:.,                   ‘,          Page 78                         /<   GAO/PEMDBO-9   Partnership   Projects   Framework


                                                                                          Society of Industrial Realtors. Strategies for an Effective Public-Private
                                                                                          Relationship in In-City Industrial Development. Vol. D. Indianapolis.
                                                                                          Washington, D.C.: 1976.

                                                                                          SRI ‘International. Promoting Local Public/Private Partnerships: A Pref-
                                                                                          ace for Federal Policy Makers. Menlo Park, Calif.: Public Policy Center,
                                                                                          June 1982.
                                               _.                                           Stegman, M: A., and 5. D. Holden. Nonfederal Housing Programs: How
                                                                    .                       States and Localities Are Responding to Federal Cutbacks in Low-
                                                                                            Income,Housing. Chapel Hil1,‘N.c.:~University of North Carolina, June
                                                                                            12,1987.                 .,
                 1                                              .,                      ,’       ,I    I,,
                                                                          :.               ,Stegman,M; A., and J.,D; Holden. “Public/Private Partnerships Promote
                                                                                            Low-Income Housing.” UrbanLand, July 1987, pp. 12-17.
                                                                                           Steinbach, Carol. “Tapping:Private Resources: The Linkage Trend.”
                                                                                           Journal of Housing; July-August 1987, pp. 111-13 and 118.
.,          >”
                       ,I..           ’
                                                         ,,               0,
                                                                               .I         Stoker; ,Rr-A. .“Baltimore: The Self-Evaluating City?” In C. N. Stone and
                                 ./                           ,,z         ‘!               H. T.sSanders(eds;). The Politics of Urban Development. N.p.: Univer-
                                                                    1                     .sity Press of Kansas, n.d.      “I

                                                                                         Streuver, C: W. “Partnerships: The Federal Withdrawal.” Journal of
                                                    ,.                                  .Housing,‘39 (May-June 198i>, 72-75.
                                                                                     ,:,.,, ,/., .,.     ‘..I    ,’ ,’ ‘, ‘,,
                                                                                         Symposium on Economic Development and the Public/Private Partner-
                                                                                        ‘ship: Journal of the American Planning Association, 53 (autumn 1987),
      ,,,                       ,,        ,’         ,,               .,          :’ ‘, 439;7,7* : :,,; 1 :               .,: ;.
.‘:         i ,,                ..                                  /~,,               ,i’
                                                                                  :I /I ,’   IS ,;,     ” ! : is       .: 8,
                                                                                         Terner, Jan Donald, ‘and TJ B. &ok. New Directions for Federal Housing
                                                                                         Policy: The Role of the States. Cambridge, Mass.: MIT Center for Real
                 !       ::
                        I,,:                                        ,.          :       .Estate iDevelopment, 1988.
                     ‘,  ..,,’                                      .,’            I I,.,       ‘3 “. !,           ,._ ”      ,’
                                                                                    1 Tobin, Gary A. YThe Public/Private Sector Partnership in the Redevel-
                                                                                         opment Process.” Policy Studies Journal, 11 (March 1983), 473-82.
                              ; d,                  ‘_                                               :i_,I ‘,      ;:       ,,’
                                                                                         U.S.,Conference of Mayors: A Guide to Triangular Partnerships Building
                                                                                         City Government: Private Sector-Neighborhood Cooperation. Washing-
                                                                                         ton, D.C.: n;d.

                                                                     ‘,        ,_
                                                                                    1     Page 79                            GAO/PEMD-90-9   Partnership   Projects   Framework

                                                                                                                               U.S. Department of Housing and Urban Development. An Impact Evalu-
                                                                                                                               ation of the Urban Development Action Grant Program. Washington,
                                                                                                                               D.C.: January 1982. ‘-’

                                                                                                                               U.S. Department of Housing and Urban Development, Working Partners:
                                                                                                                               100 SuccessStories of Local Community Public Private Partnerships at
                                                                                                                               Work. Washington, D.C.: March 1984.

                                                                                                                              U.S. Department of Housing and Urban Development. Working Partners
                                                                                                                              1984: .Reports of Successful.Local Community Development Public/Pri-
‘I’                                                                                    ;’
                                                                                                                              Vate’Partnerships. Washington; DC.: April 1984.
                                                                                                                                                         ,( :
                                                                                                                              U.S. Department of Housing and Urban Development. Public-Private
                             .             .               “’                                            ‘5                   Par!tnerships,Leveraging YourCDBG. Washington, D.C.: 1984.
                                                                                                                              ‘,      ,,          : ‘.“i
                                                                                                                              US Department of Housing and Urban Development. “The
                                                                                                                              Entrepreneurial American City.” Conference report, Tampa, Florida,
                                                                                                                              February 9~11,.l,987.~~~~~~:I
                        ,:                                      :
                                                                                                                              U.S Department :of%Housingand Urban Development. Outstanding Local
..’                                                                                                                           Partnershipsin Community Development Programs and Projects:
                                                                                                                              National Recognition :Program for,Community Development Excellence.
                                                                                                                              Washington, DC.: March 1987.
                                                                                            ,.I,                                                           :.        ,.

                                                                                                                              U.S. Departmentof Housing and Urban Development. An Evaluation of
                                                                                                                              the Neighborhood Development Demonstration. Washington, D.C.: July
      :.                                                        . ‘,,                        ,:’     ,                         1988. ‘:a         8,’ i
              ,, ;
              ,,    ;                 _’
                                      _’                          /,, I
                                                                           I                   .-_ ,,/‘,
                                                                                                   .-_        ,,/   ‘,           ‘A‘ Z,,,’ I 1.) ,: .” ;! “.
                                                                                                                                   ‘A‘       Z,,,’   I   1.)    ,:        .”   ;!   “.

                                                                                                                              US ‘General Accounting Office! Criteria for Participation in the Urban
                                                                                                                              Development Action Grant Program Should Be Refined, ~~~~-80-80. Wash-
.,                 ‘, .,                       ,’                           .j i.: :.*                                        ington,~QC;:    :$980. j .‘:‘I , ,c
 ‘,‘,                 :,:,                                      ,’         ,‘I,: ’8
                                                                           ,‘I ,:                                                         .;“: ‘/‘/ ,tc:,
                                                                                                                              )- ;; .,., .;“:
                                                                                                                              )-                       ,tc:, ,.;,.;::11 .,,.,, ,j,.
                                                                                                                              US. General A&ounting:Office. STheCommunity Development Block
                                                                                                                              Grant Program Can Be More Effective in Revitalizing the Nation’s Cities,
                                 :.                   I’                                                                      GAoj~C~D-81-~6.~Washington,~D.C~:~Aprill981.
                                                                    .’                                                                                   ,’ I
                                                                                                                              U.S. General Accounting Office. .LessonsLearned From Past Block
                                                                                                                              Grants: Implications for Congressional Oversight, GAO/IPE~~-8. Washing-
                                                                                                                              ton,.D.C.: September 1982:
                                                                                                                              U.S. General Accounting Office. Insights Into Major Urban Development
                                                                                                                              Action Grant Issues, GAO/RCED84-66. Washington, D.C.: March 1984.

                                                    ..,                        )‘/‘:                                          Page 80                                                    ,GAO/PEMD-90-9   Partnership   Projects   Framework


                                                                                              U.S. General Accounting Office. Estimated Employment Effects of Fed-
                                                                                              eral Economic Development Programs, GAO/OCE-84-4. Washington, DC.:
                                                                                              August 1984.
.’                             ,.         ,.                                                  U.S. General Accounting Office. Federal Economic Development Assis-
                                                                                              tance Programs: An Econometric Analysis of Their Employment Effects
                                                                                              1974-78, GAO/OCE84-5. Washington, DC.: August 1984.

                                                                                              U.S. General Accounting Office. Block Grants Brought Funding Changes
                                                                                              and Adjustments to Program Priorities, GAO/HRD-85-33. Washington, D.C.:
                               ,                                                              February .1985.,        x,

                                                                                             U:S. General Accounting Office.The Urban Development Action Grant
                                                                                             Application Selection System: Basis, Criticisms, and Alternatives, GAO/
                                                                                             RC~~-85-77. Washington, D.C.: March 1985.

                                                                                             U.S. General Accounting Office. Housing Allowances: An Assessment of
                                                                                             Program Participation and Effects, GAO~PEMD-86-3.Washington, D.C.: Feb-
                                                                              , ‘.           ruary 1986.
       ,)                 I,

                                                                                             U.S..General Accounting Office. Case Study Evaluations, transfer paper
                                                                                             9. Washington, DC.: April 1987.
                                                                                 U.S. General Accounting Office. Children’s Programs: A Comparative
                                                                                 Evaluation Framework and Five Illustrations, GAO/PEMD-~~-~~BR. Wash-
; ,, t,.,‘,., ‘. ,i                                                      .” ! : ington,:D.C.:  August 1988.
                                                                                           ..,:/:” ,
                                                                                 U.S. General Accounting Office. Housing and Urban Development Issues,
      I         ”,             ,’         * ‘.:                              :, .GAO/OGC-89-22TR.Washington, D.&November 1988.
                  ,I !              :*         >>’         !’              :,,:,i,.:;.,           d;.‘:‘~;   I?‘,       , ‘,;$   ,‘,/.   I’,

                                                                                              U.S. General A&o.unting’Office. Urban Development Action Grants:
                                                                                              Effects of the 1987 Amendments on Project Selection, GAO/RCED-89-64.
     ,;,        /,, ,,                   x:.: :,                    ',       ?            s, Washington, D.C.:,January ,1989.,‘.’
                                                                                                        I ‘,’                 I, ,,.
                                                                                              U.S. General Accounting &ice. Urban Action Grants: An Analysis of
                                                                                              Eligibility and Selection Criteria and Program Results, GAO/RCED-89-143.
                     ,.                              ,_'                                    ‘,-Washington,    DC.: Julytl989.
                                                                                              U.S. General Accounting Office. Partnership Projects: Federal Support
                                                                                              for Public-Private Housing and Development Efforts, PEMD-89-26FS. Wash-
                                                                                              ington, D.C.: September 1989.

                                                                ’                            Page 81                                           GAO/PEMD-90-9   Partnership   Projects   Framework

                                                     United Way of America. Practical Advice for Community Deci-
                                                     sionmakers on Assessing and Addressing Community Needs. Alexan-
                                                     dria, Va.: nd.

   *       ”                                         Urban Institute. Performance Measurement: A Guide for Local Officials.
                                                     Washington, D.C.: 1980.

                                                     Urban Institute. “Privatization of Public Services: A Cautionary Note.”
                                                     Policy and Research Report. Washington, D.C.: winter 1988.

                                                     Vidal. Avis C., Arnold M. Howitt, and Kathleen P. Foster. Stimulating
                                                     Community Development: An Assessment of the Local Initiatives Sup-
                                                     port Corporation. Cambridge, Mass.: John F, Kennedy School of Govern-
                                                     ment,.Harvard University, June 1986.

                                                     Walker, J. “Privatization of Housing Programs: Policy and Implications.”
                                                     Journal of Housing, November-December 1986, pp. 241-50.

                                                     Weaver, Clyde, and Marcel Dennert. “Economic Development and the
                                                     Public-Private Partnership.” Journal of the American Planning Associa-
                                                     tion, 53 (autumn 1987), 430-77.

                                                     Weaver, E. T. “Blurred Public/Voluntary Roles.” The Bureaucrat, spring
                                                     1983, pp. 15-16.

                                                     Webb, Eugene J., et al. Unobtrusive Measures: Nonreactive Research in
                                                     the Social Sciences.Chicago: Rand McNally, 1960.

                                                     Whitten, N., aXWolfe, A. “Network Analysis,” pp. 717-46. In D. D.
                                                     Honigman (ed.). The Handbook of Social and Cultural Anthropology,
               :.          ;.’
                                                     Chicago: Rand-McNally, .1974.
                    .‘,,         .:.‘,          ‘,                           :/ ,
                                                     Wholey, Joseph S. Evaluation:‘Promise and Performance. Washington,
                                                     D.C.: Urban Institute, 1979.

                                                     Wilmoth, D. “Regional Economic Policy and the New Corporatism.” In L.
                                                     Sowers and W. Tabb (eds.). Sunbelt/Snowbelt Urban Development and
                                                     Regional Restructuring. New York: Oxford University Press, 1984.

(973660)                                             Page 82                           GAO/PEMD-90-9   Partnership   Projects   Framework