Improvements Needed in the Administration of Contracts for Evaluations and Studies of Antipoverty Programs

Published by the Government Accountability Office on 1971-12-28.

Below is a raw (and likely hideous) rendition of the original report. (PDF)


Improvements Needed In
The Administration Of Contracts
For Evaluations And Studies Of
Antipoverty Programs B.73o515
Office of Economic Opportunity


           10954%          1     DEC. 28, a 97 2
                          COMFTROLLER     GENERAL     OF      THE       UNITED     STATES
                                        WASWINGTON.    D.C.         20548

      B- 130515

      To the      President       of the Senate     and the
.?’   Speaker       of the     House    of Representatives

                This is our report      on improvements       needed    in the ad-
      ministration        of contracts   for evaluations    and studies      of anti-
I     poverty      programs      of the Office  of Economic     Opportunity.                                3’   -3

                 Our     review   was made pursuant    to the Budget                              and Ac-
      counting         Act,   1921 (31 U.S.C. 53), and the Accounting                                and
      Auditing         Act of 1950 (31 U.S.C.    67).

             Copies    of this report     are being sent to the Director,
      Office  of Management         and Budget,    and to the Director,   Of-
      fice of Economic       Opportunity.

                                                              Comptroller                   General
                                                              of the United                 States

                                SOTH ANNIVERSARY                     1921-       1971

DIGEST                                                               1


       1    INTRODUCTION                                             5
                Scope of review                                      5

            USEFULNESS                                               7

            USEFUL STUDIES                                          11
                Need for clearer     contract    objectives
                  and work descriptions                             11
                Need for improvements in monitoring         con-
                  tractor    performance                            14
                     Need for monitoring      guidelines            14
                     Need for effective      action on con-
                        tract problems                              15
                     Need for continuity      in monitoring         18

            USE OF STUDY RESULTS                                    20

            COMMENTS                                                22
               Conclusions                                          22
               Recommendations to the Director, OEO                 23
               Agency comments                                      23


        I   Letter dated July 29, 1971, from the Deputy
               Director, Office of Economic Opportunity,
               to the General Accounting Office                     25

   II       Summary information     on selected   contracts         28

 III        Principal   officials of the Office of Eco-
               nomic Opportunity  responsible  for activi-
               ties discussed in this report                        29

GAO     General      Accounting   Office

OEO     Office    of Economic Opportunity

VISTA   Volunteers      in Service   to America
                                                      ANTIPOVERTY PROGRAMS
                                                      Office  of Economic Opportunity B-130515

I           The law directs         the Office    of Economic       Opportunity       (OEO) to provide        for con-
I           tinuing    evaluations       of antipoverty       programs,     including     their    effectiveness
I           in achieving       stated    goals.     OEO may, for this         purpose,    contract      for indepen-
I           dent evaluations          of programs     or projects.       The law also authorizes             OEO to
            contract     for studies       to test or assist        in the development          of new approaches
I           or methods      to further       the purpose    of antipoverty        programs.

1           In carrying    out its functions,             OEO has been a pioneer   in advancing                the state
I           of the art of evaluating       social          action programs,  an important    but              complex and
I           difficult   task.
             During    fiscal    years    1968 through   1970, OEO entered      into 237              evaluation    and
I           study     contracts     amounting    to about $30 million:       The General                Accounting   Of-
I            fice    (GAO) examined       the management    of these contracts       because             it wished to
I            determine      whether    useful  results   had been obtained,       because             there    was a large
I            amount of money involved,           and because there     was substantial                congressional      in-
I            terest    in OEO's efforts.


            GAO reviewed    14 contracts        amounting      to $3.2 million,    completed              or scheduled
            to be completed     during     fiscal     year 1970.      Of these contracts,               10 did not
            fully  achieve   their     purpose--     to provide    OEO with objective       and           useful  informa-
            tion to aid in the design,            development,     and assessment     of its            programs.

            OEO considered     the contractor's        reports    under eight       of the contracts       as of
            no use or as falling       short    of intended      objectives.       Although     OEO considered
            the reports    under the six other         contracts     adequate     and useful,      GAO believes
            that the reports      under two of these contracts               were of questionable       value be-
            cause the contractors'        independence       may have been compromised            due to their
            involvement    in the operations        they were evaluating.             (See p. 7.)

             GAO believes     that   several      areas    of       OEO's contract   administration          need   im-

            Need for clear       contract       specifications

            Contract    specifications          did not always      clearly   and accurately    describe the
            technical    requirements          of services     to be procured,     contrary  to the Federal
            Procurement     Regulations.           The studies    resulting    from two contracts    were not

     --      Sheet                                              I
               Prior    to the completion         of GAO's review,       OEO issued    instructions      aimed
               at tightening         the administration        of its contracts.       The new instructions
               should     improve      the management     of OEO's evaluation       and study contracts,                   but
               GAO believes        that additional      action    is needed to effectively          deal with
               the issues       discussed    in this    report.      (See p. 22.)

                   OR SUGGES!UONS

               OEO should:

                    --Provide   guidelines    to assist    OEO employees             in selecting         contractors
                       that have the ability      to make the required               evaluations         and studies
                       in an objective     and independent    manner.

                    --Issue      detailed      guidelines     for   use by OEO project         managers      in   monitoring
                        contractors'        performances.

                    --Provide    guidance     and training          to project    managers      to     increase    their
                       abi'lity  to effectively       resolve         problems   encountered         in contractor       per-

                    --Ensure    continuity          in monitoring      contractor performance    by assigning
i                      project   managers         to replace      those who leave the agency or who are                  re-
 I                     assigned    before       the work under the contract        is completed.
 I                  --Develop     and implement       a system to ensure       (1) the formal     assessment         of
;                      the results      of evaluations      and studies     and (2) the effective        utilization
                       of the results      of contractors'       studies    to improve     the designs     and adminis-
I                      tration    of programs      or to devefop      new program    approaches,     when appro-
                       priate.      (See p. 23.)

                OEO acknowledged     the problems   discussed     in GAO's report      and agreed with the
                intent   of its recommendations.       OEO stated    that a high-level      task force   had
                been convened which would deal with those problems            identified      in GAO's report
                that   had not already   been corrected.       (See app. I.)


                The Federal      Government  is making increased       use of private      contractors      to
                evaluate     and study its programs     and related      activities.     This report     illus-
                trates    the need for exercising     careful    control      over such contracts      to en-
                sure that     the results   obtained  are objective       and usefuJ    and are effectively


     Tear   Sheet                                           2 Y-3

       The Economic Opportunity  Act of 1964, as amended
(42 U.S.C. 2701),commonly called the war on poverty act,
authorizes   the Office of Economic Opportunity     to enter into
contracts   with public agencies and private    organizations
and persons to assist in carrying    out the various require-
ments of the act,

        Section 233 of title     II of the act, which authorizes
OEO to fund Community Action Programs and related               programs,
directs    OEO to provide for continuing       evaluations      of the
programs and authorizes       OEO to contract      for independent
evaluations     of (1) their effectiveness       in achieving      stated
goals, (2) their impact on related         antipoverty      programs,
and (3) their structures       and mechanisms for the delivery            of
the required     services.    Section  232  authorizes      OEO  to   con-
tract for other studies to test or assist             in the develop-
ment of new approaches or methods that will aid in over-
coming special problems or otherwise          aid in furthering        the
purposes of title       II of the act.

     During fiscal    years     1968 through 1970, OEO awarded
237 evaluation    and study     contracts  in the total amount of
about $30 million.


        Our review was directed      toward determining      whether the
policies,    procedures,    and practices     followed by OEO in ad-
ministering     evaluation    and study contracts      were adequate to
ensure that OEO was obtaining         objective     and useful informa-
tion under these contracts        and that OEO was making maximum
utilization     of final   study reports.

       We selected for review 14 evaluation  and study con-
tracts   totaling  about $3.2 million, which had been recently
completed or which were to be completed during fiscal     year

       Our selection  of these contracts        was based on a con-
sideration   of contracts     with high dollar values,      contracts
that had been completed or that were to be completed during
our review, and contracts        representing     a cross section of
the various antipoverty       programs administered      by OEO but
did not include 34 contracts         for studies of Job Corps and
Head Start projects,      since   these  projects    were no longer
under OEOss management.

      We reviewed the records for the 14 contracts      and dis-
cussed them with present and former agency employees and
with some of the contractors   to determine whether the prod-
ucts produced or being produced by the contractors       were
acceptable   and useful to OEO. We also solicited      comments
from 10 of the 14 contractors.     The comments received were
considered by us in the preparation    of this report.


        OEO has been a pioneer in advancing the state of the
art of evaluating     social action programs, an increasingly
important     but complex and difficult     task.    OEO's purpose in
entering    into evaluation     and study contracts    is to obtain
useful and objective      information    to aid in the design, de-
velopment,     and management of its antipoverty       programs.    Our
reviews however9 showed that, under several of these con-
tracts,   this purpose was not fully       accomplished    in that the
reports    submitted by the contractors       were not useful or
were of limited     use for OEO program operations,

      We examined 14 contracts--eight      awarded in fiscal      year
1968 and six in fiscal   year 1969--to determine the adequacy
of OEO's administration    of the contracts     and its utiliza-
tion of the study reports.      Our interview    of OEO officials
and our review of available     contract   records and related
correspondence   for the 14 contracts     showed that:

      --OEO considered the study reports          obtained under six
         of the contracts,      five of which were awarded in fis-
         cal year 1969, to be adequate and useful,           Webelieve,
         however, that the reports        obtained under two of the
         six contracts    were of questionable      value, because
         the contractors'     independence may have been compro-
         mised due to their       involvement   in the operations
         they were evaluating.

      --QEO considered the reports obtained under the remain-
         ing eight contractsp  seven of which were awarded in
         fiscal  year 1968, as of no use or as falling  short
         of meeting the intended objectives,

      Comments regarding     these eight   contracts    follow.

     --A former director,        Office of Special Field Programs,
        criticized      as being unconvincing   and lacking in ba-
        sic statistical      credibility   an evaluation   costing
        $179,996 of the OEO migrant adult education program.
        He stated that "The work done to date lacks so much

             credibility     that it would never convince      our grantees,
             our staff,    the public and fee-X-*

       --A  report on a study of planning development in Com-
         munity Action Programs made for the Office of Pro-
         gram Development at a cost of $151,166 was returned
         to the contractor     for major revision    at no additional
         cost to OEO. According to OEO officials         the report
         failed   to fulfill   the requirements    of the contract.
         Even after revisions      were made, the Chief of the
         Evaluation   Division    commented that the report still
         contained "too much description        and not enough evalu-

       --A study,costing       $285,7489of Volunteers    in Service to
          America recruiting       methods was considered of such
          poor quality     that it served as a basis for rejecting
          the contractor's      proposal for a later contract     to do
          VISTA recruiting.        The VISTA review panel reported:
                   'Yt'he final  research report last year con-
                   tained little     more than a lengthy list     of
                   platitudes    already known to Volunteers      in
                   Service to America's most junior      recruit-
       --A study, evaluation,      and training    contract    in the
          amount of $383,276 for OEO's Office of Public Affairs
          was closed out at the request of the contractor            be-
          fore it had submitted a final report.           According to
          the Director  of Procurement,      problems which arose
          between the contractor     and the OEO project      manager
          during the administration      of the contract     could not
          be adequately resolved.      The contract     was terminated
          on January 5, 1971; the contract        amount was reduced
          by $8,000,

      Reports or information       pertaining  to the other four
contracts   were similarly     cited by OEO officials     as lacking
in expected quality;      that is, as being too general or as
being of limited   usefulness      because contract   objectives    had
not been clearly   defined.

       In commenting on a draft of this report,    four contrac-
tors   took exception to OEO's evaluation  of their work and

two did not take exception to OEO's evaluation  of their                     work,
Two contractors did not comment on our draft report.

       The reports   on four of these eight contracts     were dis-
tributed   to OEO program employees,      Final reports for two
other contracts     were not distributed;   another contract   was
terminated    prior to its intended completion date, for which
the contractor     submitted only a status report summarizing
its progress to the date of termination;       and the remaining
contract   was completed without the contractor's      issuing a
final report.
        In addition     to OEO's questioning       the above eight con-
tracts,      we questioned the value of the work performed and
the resultant       reports under two contracts           that OEO consid-
ered adequate and useful,          because the contractors            were
either participating         in the operations       of the OEO program
which they were evaluating          or providing      technical        aid to
certain      of the projects     under the programs.           Highlights      of
our observations        on these two contracts        follow.
         -0EO awarded in March 1969 a contract               in the amount
          of $100,350 for the evaluation          of VISTA programs to
          a contractor      which at the time had contracts               of about
           $5 million    under which he was carrying            out certain
          operations     for VISTA.      In commenting on our draft re-
          port, the contractor        stated that the consultants              who
          had performed under the contract            were independent
          agents and that there was no question as to the ob-
          jectiveness      and usefulness     of the study reports.
          Although the effect       that these other contracts               had
          upon the judgment of the consultants               employed by the
           contractor    could not be determined,          we believe that,
           to ensure objective      and useful study reports,               OEO
           should select contractors        for evaluations          that are
           independent of the programs to be evaluated.

       --Under a contract     awarded in October 1967 in the amount
          of $440,734 for OEO's Office of Program Development,
          a contractor   was required   to provide technical    assis-
          tance to rural research and demonstration      projects
          and at the same time to monitor and evaluate the
          projects.    Both the Assistant   General Counsel and the
          Chief of the Rural Branch agreed that the contractor
          should not have been allowed to determine the

technical  assistance  needed and to be provided to the proj-
ects and also to monitor and evaluate the technical       assis-
tance it provided.    Tn our opinion,  these are requirements
which, when not met independently,    represent a conflict     of

       In April 1970 OEO issued instructions             which included
guidelines    for preventing    conflict-of-interest          situations
in contracting.      The guidelines      deal principally        with pre-
cluding contract     awards to organizations          that employ former
OEO employees.     They do not, however ) provide guidance to
assist contract    planners in selecting          contractors     that can
do the required    contract   work independently          and objectively
and that are not otherwise       involved in OEO activities             which
could impair this ability.

       In commenting on a draft of this report,         OEO stated
its belief      that,    to judge the value of performing    evaluation
studies,     the cost of the studies should be compared with
total    program expenditures.       In OEO's opinion,    it is con-
ceivable that the beneficial         impact of one successful     evalu-
ation could more than justify          the overall expenditure    of
its evaluation        efforts.

      We recognize that some evaluations       may result          in greater
program benefits    that others,    but we believe that           successful
evaluations   cannot justify    defects in others which            may have
been caused by weaknesses in the administration          of        the eval-
uation contracts.

     Additional   information    on the 14 contracts considered
in our review,  including     a summary statement on the useful-
ness of the studies,    is shown in appendix II of this report.



                        IN MORE USEFUL STUDIES

       Frocedural   problems weakened OEO's control    over the
work done under the contracts       we reviewed and lessened the
likelihood    of receiving   useful study results.

       --Contract  objectives    and statements of work should
          have been more clearly    defined.   Vagueness in objec-
          tives caused misunderstandings     between project man-
          agers and contractors.

       --The methods used to monitor studies were left largely
          to the preferences   of individual  project    managers.
          Contract problems were not identified       and resolved
          in an effective    mmer.


        The Federal Procurement Regulations,             under which these
contracts    were negotiated,        require that all parties           to a
contract    obtain a good understanding           of its objectives         and
scope.     Implicit   in   this   requirement     is  the   need   for    agency
employees to adequately define contract               objectives       and to
draft contract      specifications       which must be met to achieve
the objectives.       According to section l-1.307-2              of the Fed-
eral Procurement Regulations,            contract    specifications        should
clearly    and accurately       describe the technical         requirements
of the products or services to be procured.

       For two contracts   the resulting   studies were not satis-
factory   because of problems in initially      determining con-
tract objectives    and formulating    work descriptions.

       In the first case OEO awarded a contract      in the amount
of $285,748 for a study of VISTA recruiting        methods.    The
objectives   of the contract   were not clearly    established
either prior to or after the contract       award.   Our inquiries
of OEO employees associated     with the contract    revealed that
there was disagreement     among VISTA officials    as to whether
the contract was for the actual recruitment   of VISTAvolun-
teers or for research or evaluation   of VISTA recruitment

         An associate    director     of VISTA, who had assumed respon-
sibility     for the    contract    after work began, told us that the
original     purpose    of contract     was to help recruit  more vol-
unteers but that,        because the then Director      of VISTA did not
favor a recruitment         contract,    it had been awarded as a re-
cruiting-research        effort.

        In a final   evaluation    of the work    done under the con-
tract,    VISTA officials     concluded that     (1) as a recruiting
effort,    it had fallen     short of VISTA's     own recruiting     re-
sults during that same period and (2)            as a research effort,
the contractorDs       final report was less      than satisfactory.

       In the other case, which concerned a contract        in the
amount of $329,355 for a study of the Community Action Pro-
gram in a selected county of Texas, the work description
was not sufficiently       detailed  to reasonably ensure that the
contract    objectives    would be accomplished.    This contract
was to be phase II of a two-phased study of the Community
Action Program in Austin and Travis County, Texas.            Both
phases were to produce results         which could be combined with
the results     of similar   studies being conducted in other geo-
graphical    areas in an attempt to develop an overall        strategy
for evaluating      OEO's Community Action Program.

        According to a former chief of the Evaluation      Division,
who had been responsible     for managing this contract,     OEO
officials     decided upon an additional   major objective   for
phase II of the study--that      the contractor  formulate    sugges-
tions for improving the Texas program.

        The work description     for phase II, however, did not
require    final    recommendations by the contractor,        contrary
to what had been contemplated.           As a result,  after the con-
tractor's     final    report was received,   the Chief of the Bvalua-
tion Division       requested a letter    from the contractor      setting
forth recommendations based upon the 3-year, two-phased
study which cost OEO about $670,000.            In its reply,    the con-
tractor    pointed out that:

      "In     atterilpting    to arrive    at policy   recommendations
      after      our 3-year     study     of the CAP [Community Ac-
      tion Program] effort,   the inescapable  conclusion
      is that,  as social scientists,   we are better able
      to define problems          than we are at recommending
      program changes."

Nevertheless   the contractor provided a list    of suggestions
and observations   which it called off-the-cuff    comments.
According to OEO officials,   these comments were not fur-
nished to persons responsible     for managing the Community
Action Program in Texas for their use.

       In April 1970, recognizing       the need for clear contract
objectives    and specific  work descriptions,        OEO issued to
its headquarters     staff instructions     requiring    that contract
work descriptions     be in such detail     that OEO could hold the
contractors    accountable   for exactly that for which OEO had


       Improved monitoring of contractor   perfo-rmance was
needed to ensure that contractors'     work was progressing

     The responsibility       for contract  monitoring     is assigned
to an OEO official      designated by the contracting        officer to
act as the project     manager.    The contracts    usually described
the project  manager and his authority       as follows.

      "A representative     of the Government, designated
      by the Contracting     Officer    ***.   He has full
      authority    to represent    the Contracting    Officer   in
      connection with the operations         under this con-
      tract,    except that, he is not authorized        to issue
      orders which change the work to be performed here-
      under, the compensation,        or the period of perfor-

       The effectiveness     of a project     manager in overseeing
and guiding the contract       work can contribute        significantly
to the success or failure        of the contractor's       efforts.
OEO's April 1970 instructions         established    the policy for
the selection     of project   managers and broadly described
their role and authority.          The guidelines,    however, did not
adequately provide for (1) monitoring           contractor      perfor-
mance, (2) dealing effectively         with problems encountered in
contractor    performance,    and (3) maintaining       continuity      in
monitoring    contracts    when project    managers responsible         for
monitoring    the contracts    left the agency or were reassigned.

Need for    monitoring      guidelines

        Effective     monitoring     requires that project      managers
keep fully       informed of contractors'      progress and any diffi-
culties     encountered in meeting contract         objectives.      Good
monitoring       generally    includes

      --visiting     contractors'        sites,

      --obtaining        and analyzing reports from the contractor
         regarding       the progress being made, and

      --maintaining   formal     records   of agreements    reached
         with the contractor.

        In the absence of specific    guidelines,   contract      moni-
toring was largely      dependent upon the preferences,        abili-
ties, and work loads of the individual         OEO project     managers.
As a result,    progress reports    in some cases were not re-
quired or did not contain meaningful         data; site visits        gen-
erally    were not made; and basic data on contractor          perfor-
mance, such as the project       manager's assessment of the con-
tractor's    work progress and achievement of interim          goals in
many instances was not available       in the contract     files.

        For example, OEQ awarded a contract    which was com-
pleted on June 30, 1969, at a cost of $179,996, for an eval-
uation of the adult education and training       program for mi-
grants.     The contract   required that progress reports be
submitted every 30 days to OEO. According to the project
manager, the reports were not adequate for use in monitor-
ing contractor     performance because they were too vague re-
garding work progress and problems encountered by the con-

       Although the project     manager considered the contrac-
tor's progress reports      inadequate,     he did not visit   the
worksites    while the contract    was being performed.       Instead,
he relied on the contractor's       initial    working plan, the
progress reports,     and some telephone communication with the
contractor.      These means of monitoring       contractor  perfor-
mance proved inadequate as the final          study report submitted
by the contractor     was considered useless by OEO officials.

Need for effective    action
on contract  problems

     In two of the contracts     included in our review, prob-
lems which adversely affected      the progress and outcomes of
the studies were not effectively      dealt with by the project

        In the case of the contract    to evaluate education and
training    programs for migrants,   the project    manager in-
formed us that in February 1969 he first         began to realize
that the contractor     was not developing    the statistical     data
required by the contract.       The project   manager considered
this data necessary to draw valid study conclusions          and to
make the final    report responsive    to agency needs.    Payments
continued to be made to the contractor,         however, during the
balance of the H-month contract        period to June 30, 1969,
by which time $167,493 of the contract        amount of $179,996
had been paid to the contractor.         In July 1969, 5 months
after the project    manager realized     the problems, the con-
tractor  was for the first    time formally     notified by an OEO
official  that the work was inadequate and was not useful to

        In our opinion,    had the project  manager been adequately
monitoring     this contract,    he might have (1) recognized at
an earlier     date that the contractor's    performance was un-
satisfactory      and (2) obtained the assistance    of other CEO
offices,     such as the Office of General Counsel or the Fro-
curement Division,       to determine whether payments under the
contract     should have been discontinued    until  the problems
were corrected.

       AS a result  of our inquiries,   the contract  was brought
to the attention    of the General Counsel of OEO and the ap-
propriate   procurement officials.     As of November 1971 CEO
was considering    what, if any,action   would be taken to re-
solve the contract    problems.

        In another case, which involved a study, evaluation,
and training      contract     undertaken for the OEO Office of
Public Affairs,       the project     manager did not ask for assis-
tance from the Procurement Division             or the General Counsel
even though problems were encountered which warranted                such
attention.      In this case the contractor          failed to provide a
full-time    project     director    and did not carry out normal ad-
ministrative      functions     to support the project.       The project
manager was not successful           in resolving    these problems, and
the contractor       eventually    contended that it was unable to
complete the work as contemplated.

     The basis for the contractor's   complaint         was detailed
in a letter  to OEO's Procurement Division.

      "Because of repeated changes of instruction  by
      the OEO Project Manager, and repeated changes in
      the conception of the project        it has proven im-
      possible for us to fulfill       all of the physical
      requirements   of the contract.        This is a re-
      quest to be relieved     of certain     requirements,
      with appropriate    adjustments,     so that the
      contract   may be terminated."

      OEO agreed to the contractor's    request,   and on Janu-
ary 5, 1971, the contract   was terminated    before the final
report was prepared.   Ihecontract   amount was reduced by

       OEOps system of making progress payments to contrac-
tors did not, at the time of our fieldwork,           require  the con-
current approval of the project manager.            Periodic  billings
were sent by the contractor       to the Defense Contract Audit
Agency1 for a routine      fiscal   check and from there were for-
warded to OEO's Finance Division         for payment.     Only the
final   payment to the contractor      required   the project    man-
ager's specific     approval which 'was to be based on the satis-
factory   completion ot the contract        work.

        The Procurement Director        of OEO informed us that a
double vouchering       system would more actively         involve project
managers in the approval of progress payments.                 Under such
a system a contractor        would send duplicate       vouchers to the
Defense Contract Audit Agency and the OEO project                 manager
for their respective        approvals.       The Defense Contract Audit
Agency would continue to be responsible             for determining     the
validity    of the contractor's        costs, and the project       manager
would certify     to the quality       of the contractor's      work.   We
believe that a system requiring            the project   manager's ap-
proval of progress payments would strengthen               his role in
monitoring    contractor     performance.

       In addition,  project   managers     needed to be better ac-
quainted with OEO's administrative          procedures for dealing
with problems involving      contractor     performance.   Project
managers generally    were chosen from       program employees having
technical   backgrounds in the social        sciences and having

1 The Defense Contract Audit Agency acts           as the cognizant
 audit agency for most OEO contracts.

little    orientation     in procurement,    financial administration,
and legal matters.          Therefore OEO, through guidelines       and
training,     should acquaint project      managers with the special-
ized assistance       available    to them in dealing with problems
of contract      administration.

Need for    continuity    in monitoring

        For the management of evaluation           and study projects     to
be effective,      OEO must ensure that project         managers are
qualified     to monitor contractor       performance of a project
and that continuity       of responsibility        for the project     is
maintained     when project    managers leave the agency or o%her-
wise relinquish      responsibility     for the project       before it 5s
completed.      In the following    two cases, OEO did not ap-
point project      managers to replace those who left.             Conse-
quently OEO was not in a position           to monitor the contract
work and resolve problems precluding             the fulfillment     of
contract    objectives.

       The first      case involved a contract           for $100,000 for a
survey of mental health care programs to which a project
manager was no% assigned during the greater part of i%s
duration,      The contract,      which was awarded on June 24, 1968,
originally     designated a project         manager, but he left OEO
soon after the award of the contract,                  A replacement was
not made by the OEO contracting             officer.        The contractor
experienced      difficulties      in doing the contract         work, espe-
cially     in organizing      and controlling       the survey,      As a re-
sult,    the contract       was in effect     for about 14 months, dur-
ing which time virtually          no work was done.           On June 26,
1969, and again, on July 14, 1969, the contractor                    requested
that the project         be terminated.       The contractor9s       July 14,
1969, request included the following                statement:
      I'**   at this point in time %he contract       project
      is not as valid as when originally       conceived.
      Therefore,    taking everything    into consideration,
      it is the opinion of the [contractor]        that it is
      not in the best interest      of the Government to
      continue the contract     and recommends that it be
In September 1969 OEO terminated      the contract. The con-
tractor claimed costs of $25,000, which, as of September
1971, were subject to a final    settlement.

      We believe that,   if an OEO project   manager had been
assigned, he would have been in a position      to recognize
the contractor's   problems and seek earlier    resolutions    of
the problems.    The unused portion   of the $100,000 earmarked
for this project   might have been released earlier       and might
have been put to use on a more worthwhile      antipoverty    ef-

       In the case of a study of the summer Youth program
which cost $100,408, OEO did not assign a new project              man-
ager after the designated project         manager left the agency.
The need for continuity       in monitoring     this study was partic-
ularly   critical   because (1) the project        manager, before
leaving OEO, directed      the contractor     to revise its report,
at an additional     cost of $2,200 to satisfycontract         require-
ments and (2) during the period of the contract,            the summer
Youth program was transferred       to a newly created OEO pro-
gram office.      As a result   of not appointing      a new project
manager, OEO lost control       over the study and considered the
final report to be of lower quality         than expected.       The re-
port, however, was used as a basis for developing new pro-
gram guidelines.


                           NEED FOR A SYSTEM
      Although the effectiveness     and usefulness   of evalua-
tions and studies undertaken by OEO contractors        depend on
the extent to which the results      of such studies can be,
and actually   are, used, we noted that OEO had no system for
ensuring that the validity     of study results    was formally
assessed and that valid contractor      recommendations were
used for the intended management purposes.

        For 12 of the 14 contracts,       final   reports were submit-
ted by the contractors;         however, most of the reports were
not formally     assessed to determine their usefulness,             Al-
though informal      appraisals    of the general acceptability          of
the reports were made by individual           project    managers, OEO
project    files  did not contain documentation          of systematic
assessments of contractors'         conclusions    and recommendations
and the reasons for taking,         or not taking,     action on them.
Further CEO did not have a system to ensure that appropri-
ate action would be taken on all study findings,               conclusions,
and recommendations consideredworthwhileor               to review im-
plementation     of any actions taken as a result          of the studies,

       OEO has recognized that there is a need for establish-
ing a system for assessing and utilizing        the results   of
study reports.      In June   1969  OEO awarded a  contract for
$26,293 for the development of a system for disseminating
the data developed under evaluations        and studies within
both OEO headquarters     and l&al     Community Action Agencies.
The request for proposals for the contract         pointed out the
following   deficiency   within    OEO.

      llCAP [Community Action Program] program evalua-
      tions and evaluation   systems have been developed
      on a piece meal basis.       These systems are provid-
      ing piece meal information      to policy makers and
      program operators.    Little    is known about what
      happens to evaluation     information    and there has
      been no assessment of the value of evaluation        in-
      formation  that has been produced or the methodol-
      ogy that has produced the infomation.t'
        The contractor   confirmed        this   deficiency   by stating   in
his   final  report that:

       "Although we certainly    don't have the data to sup-
       port any generalizations,     our interview   did rein-
       force the impression that utilization       of evalua-
       tion results  in general was mostly an accidental
       consequence of an interest     in a program somewhere
       in OEO and information    about that program appear-
       ing somewhere else within    the Agency."

      The contractor   recommended an extension of its con-
tract for t'he purpose of developing a prototype        handbook
for use by OEO officials    involved in initiating,      monitoring,
and assessing evaluative    research efforts;     a prototype    pam-
phlet for use by potential     contractors   and grantees in pre-
paring proposals and interim     and final reports;     and a pro-
totype brochure for non-research-oriented        employees explain-
ing in simple terms the basics of research in antipoverty

         In April 1970 OEO awarded the contractor            a follow-on
contract     in the amount of $30,967 to develop for use by OEO
officials      standards and guidelines         for preparing,    monitoring,
and assessing evaluation          projects.       In March 1971 OEO re-
ceived the contractor's         final report and as of September
1971 was utilizing        the report in developing new procedures
and regulations       designed to increase the effectiveness             and
utilization       of OEO evaluation      projects.

        The final   assessment of study results         and, where ap-
propriate,     the issuance of directives         for implementing      con-
tractor    recommendational        level above the project       managers.
OEO should consider using panels of top-level               officials,
such as individual       office      and program directors     and assis-
tant and associate       directors,      who would have responsibility
for formally      assessing and, where appropriate,          initiating
action on contractors'          recommendations.     The panel approach
might utilize      one of the several boards which OEO has estab-
lished to review proposed procurement actions and to evalu-
ate contractor      proposals.



      OEO needed to make substantial        improvement in the
planning and monitoring      of evaluations    and studies to ob-
tain useful   information    commensurate with the high cost of
these evaluations     and studies.

       Prior to the completion        of our fieldwork,       OEO developed
instructions     aimed at tightening       the overall      administration
of its contracts      and grants,      Some   of the   instructions,       is-
sued in April      1970, deal with contract       planning,      the prep-
aration     of work descriptions,      the  use  of  OEO    review   panels
to consider contract       proposals,     and the preparation        and
maintenance of contractor         performance records.          Other in-
structions     deal with avoiding conflicts         of interest      and
strengthening      the functions    of OEO project       managers.

        The new instructions,     if properly   implemented,      should
improve the management of OEO contracts          for evaluations        and
other studies;     however, the instructions       do not provide needed
guidelines    for (1) ensuring the independence and objectivity
of contractors,      (2) adequately monitoring      contractor     per-
formance,    (3) dealing effectively      with problems encountered
in contractor     performance,    or (4) maintaining     continuity      in
monitoring    contracts.      Special emphasis is needed in these
areas of OEO contract        management to correct    the deficiencies
identified    during our review and to prevent        them from re-

      Also, although evaluations      and studies were to give
OEO information    it needed for improving antipoverty       efforts,
OEO had no system to (1) assess the usefulness        of contractors'
conclusions    and recommendations,    (2) initiate action to
bring about needed changes, or (3) follow up on actions
taken as a result     of studies to determine whether the ac-
tions had been properly      implemented and were effective.

              TO THE DIRECTOR, OEO

     We recommend that      OEO:

     1. Provide guidelines      to assist  OEO employees in select-
        ing contractors     that have the ability    to make the
        required   evaluations    and studies  in an objective   and
        independent manner.

     2. Issue detailed       guidelines     for use by OEO project
        managers in monitoring          contractors'      performances.
        The guidelines       should require      project    managers to
        visit   contractors'      sites to observe performance,
        obtain progress reports,          and fully      document agree-
        ments or other understandings            reached between the
        project   managers and the contractors.

     3. Provide guidance and training            to project     managers
        to increase their        ability   to effectively       resolve
        problems encountered          in contractor      performance.
        Project     managers should be instructed            to seek the
        assistance     of OEO contracting,         legal,    and other
        specialists     as needed to safeguard OEO's contractual
        rights    and interests.

     4. Ensure continuity      in monitoring   contractor performance
        by assigning   project     managers to replace those who
        leave the agency or who are reassigned before work
        under the contract       is completed.

     5. Develop and implement a system to ensure (a> the
        formal assessment of the results           of evaluations    and
        study contracts--      possibly    by panels of top-level      OEO
        officials--    and (b) the effective      utilization     of the
        results     of contractors'     studies  to improve the de-
        signs and administration         of programs or to develop
        new program approaches,         when appropriate,


       OEO, in commenting on a draft of this report  in a letter
dated July 29, 1971 (see app. I), stated that it recognized
the problems discussed in the report   and concurred   in the
intent   of our recommendations.

     OEO stated also that the following        actions had been
or were being taken to improve contract        management.

     1. A high-level    task force on program management, con-
        tracting,    and grant issuance was convened to deal
        with those problems identified     in our report and in
        other OEO studies that had not already been corrected.

     2.   More attention    was being given to source selection
          criteria   and methodoloa     to ensure that,    after price
          and all other factors     were considered,    the best
          possible  contractor   was chosen.

     3. Attempts were made to phase out many demonstration
        and other small projects    which should reduce the
        number of evaluation   studies associated     with those
        projects and which should thereby reduce the work-
        load of the limited  staff    of the Evaluation    Division.

     4. Additional   employee slots were acquired to increase
        the size of the evaluation    staff,  and additional
        positions  for this staff   were requested.

                                                                                        APPENDIX I

                                                        EXECUTIVE   OFFICE     OF THE   PRESIDEW

OFFICE OF ECONORIIC                                                   WASWINGYON.        D.C.   20506

                                                                        July     29, 1971

  JIJL 29 1971

  Mr. Henry Eschwege
  Associ'ate Director
  Civil Division
  U. S. General Accounting   Office
  Washington,    D. C. 20548

  Dear Mr. Eschwege:

  Thank you for the opportunity        to comment on the draft report,            "Stronger
  Management Needed to Increase the Effectiveness              of Evaluations      of Studies
  of Anti-Poverty    Programs."     The report is interesting           and useful to us
  and other agencies engaged in performing           evaluations      of social action
  programs.     The problems discussed      are real and we concur with the intent
  of the recommendations.       As noted in the report,        we had already taken
  action to eliminate      some of the problems.        I have recently       convened a
  high level task force on Program Management, Contracting                 and Grant
  Issuance whose charter      is to deal with remaining         problems identified         in
  your report and in other studies we have made. Your report                   should have
  some additions    to avoid misinterpretation        and to increase       the usefulness
  of the recommendations      to those others who will use it to deal with
  problems in management of evaluation         studies.

  Your review of our evaluation       studies was directed      at,improving       them: We
  are concerned,     however, that the results      are subject     to misinterpretation.
  To some extent this problem cannot be avoided.            It could be easier for
  responsible    people to understand     the context   for evaluation       and the
  difficulty   of drawing conclusions       about its value from studies         such as
  yours, if your report pointed out such things as the following:

        a)   From a contract      management point of view, the number of
             inadequately    managed contracts        is important.      In judging  the
             value of performing      evaluation      studies,   however, the cost of
             the studies has to be compared with total program expenditures.
             It is entirely     conceivable      that the beneficial       impact of one
             successful    evaluation    could more than justify         the overall
             expenditure    of our evaluation        efforts.

        b)   Since many readers areilikely        to generalize     from your results
             in spite of whatever cautions        you might put in, it is desirable
             to balance the generally      negative   picture    that the numbers in
             their current    form represent    by pointing     out that four of the six
             contracts    let after June 1968 were considered          to be acceptable    by
             GAO, and one more was considered        useful by the Office of Economic
             Opportunity.     None let before July 1968 were considered           acceptable
             by GAO.


      c>    One of the reasons that good evaluations          are difficult      to per-
            form is that it is almost impossible        to find a “normal”       period
            in the operation      of a program.    You were forced to evaluate          the
            program as it existed       at the time that you conducted         your
            evaluation.     Nevertheless,     it should be recognized       that 1969
            represented   a year of very rapid change both in philosophy               and
            in personnel,     even by our standards,      This change was probably
            responsible    for at least some of the discontinuity           in management.

       d)   We understand      that Job Corps and Head Start studies              were not
            reviewed because the programs are no longer managed by the
            Office    of Economic Opportunity.            Spinning     off these programs
            reflected    a judgment      on program and management maturity.               As
            one indicator      of this,      both programs had quite well developed
            and systematized        evaluation      programs.     Since the 235 contracts
            and funding    cited appear to include            contracts    let by Job Corps
            and Head Start,       the reader needs to be cautioned             about the
            effect    on any generalization          from your results       to an overall
            measure of evaluation          contract     success.

 Your report     recommends the institution       of a number of improved management
 procedures.       There is no question     that such a system helps to improve
 the quality     of evaluation     studies.    It is, however,   of secondary  importance,
 as compared to the existence          of an appropriately   qualified   staff whose
 workload    is limited     so that there is adequate time for them to do a good
 job.    In particular    :

       4    The comparatively          rudimentary       state of development         of evaluation
            measures makes it very difficult                  to design a study which will
            yield useful       results     in areas of immediate          interest     to decision
            makers.     The people who do this work should be trained                       and
            experienced      analysts      with knowledge of social            action    programs
            and research       procedures      relating       to the social      sciences.
            Because such people are very scarce,                   one can, instead,        use teams
            of at least one experienced               analyst     with statistical       or operations
            research    training,        one researcher        with appropriate       social    science
            skills   and additional          consultants       where necessary.

      b)    Many of the studies         are sufficiently        large in scope to require
            considerable       management competence on the part of the project
            manager.       This competence is most likely             to be developed      through
            prior management experience,             For a variety       of reasons,     in-
            cluding     the need for training;         for continuity      during    vacations,
            illness     or departure     of the project       manager; and because people
            appear to work more effectively              in teams, at least two pepple
            should be assigned        to every study.         At least one of them should
            be an experienced        manager.     The number of studies          which can be
            effectively      assigned    to an individual         depends on their     scope, but
            should not exceed two or three.                If the study is complex and
            important      enough, it should be the only assignment               for the team.

                            Effective            Cost               Responsible                                                                       Ilseful,;e>s  of c,,d product
 ( 1 i)        _?Ys            dates          (-a)                   OEO office                  Purpoze                                                   and other <onmeuts
    A      Flxed    price     10-18-67        S 440.734         Proeram       Devel-    Evaloatio"      alld fea-            The reports   were considered         useful    by bol?e OEO officials.
                                                                opment                  sibility     studies       of        In our opinion,      however,    their    objectivity      was question-
                               5x70                                                     rural    research      and           able because the contractor           was required       Lo monitor    and
                                                                                        demonstration        proj-           technically   assist     the same projects.           According     to the
                                                                                        ects.                                OEO General Counsel,        these requirements         conflicted.
    B      cost plus            2- 8-60          285,748        Volunteers              Evaluation        of VISTA           The final  research      report was considered           useless    by OEO
           fixed fee                                            in Service              recruiting        methods.           officials  and was described      as informaria             already   known
                               9-::-60                          to America                                                   by VISTA junior     recruiters.
    c      Cost plus           4-15-60           329,355        Planning,               Study and evaluation                 The COI~S~IISUS of OEC! officials     contacted    was that the re-
           fixed  fee                                           Research        and     of Community Action                  port was of limited    usefulness.       The report,      along with
                              12-:00-69                         Evaluation              Program in Austin                    other Community Action Program evaluation            studies,   was
                                                                                        and Travis  County.                  submitted   to the University      of Wisconsin    for further      re-
    D      Cost plus           6-10-68           100,408        Program       Devel-    Evaluation   of summer               The last of two re"orts     was revised    twice at additional
           fixed  fee                                           opment                  Youth programs.                      cost to OEC. The 'final version       was considered     of less
                                5-::-69                                                                                      than expected   quality  by OEO officials      but reportedly    was
                                                                                                                             useful  in developing   program guidelines.
    E      Cost plus           6-24-68           100,000        Health     Affairs      Survey of psychiat-                  The contract     was terminated,      and a status   report    was sub-
           fixed  fee             tcl                                                   ric mental health                    mitted   by the contractor       which was, according       to OEO of-
                               4-u-70                                                   care programs.                       ficials,    of limited    usefulness.     Final contract      settle-
                                                                                                                             ment was still      pending as of September 1971.
    F      Cost plus            6-29-68          179.996        Program       l?evel-   Evaluation    of adult               The report    was considered       useless   by OEO officials       be-
           fixed fee                                            opment                  education    and train-              cause the contractor         did not satisfactorily       develop    the
                                6x69                                                    ing programs    for mi-              required   statistical       data.    The reported    conclusions       and
                                                                                        grants.                              recommendations,       therefore,     were considered     untenable      by
    G      Fixed    price       6-28-68          362,069        Program       Devel-    Evaluation        and re-            Five of the six final      reports    were characterized        by OEO
                                                                opment                  search.                              officials     as marginal  in quality      and as lacking      in depth
                                7-::-69                                                                                      and perception.       The usefulness     of the information       was
                                                                                                                             reported     by OEO as being minimal,       and the contractor      was
                                                                                                                             cited     as being ""able  to provide      the quality    of work ex-
     H     Fixed    price       6-29-68          383,276        Public       Affairs    Study,   evaluation,                 There was "a final         report--due    to OEO project       management
                                                                                        and training      program            difficulties,        OEO terminated    the contract       at the request
                                1x70                                                    and production       of              of the contractor        before all contract        requirements     were
                                                                                        16 mm films.                         met.       The contract    amount was reduced by $8,000.
     I     Cost plus          lo-      l-68      471,774        Planning,               Evaluation        of Cornmu-         The final     report   was considered    good by OEO and was used
           fixed fee                                            Research        and     nity Action        Pro-              as a basis.      along with reports    of similar    contracted
                                7&69                            Evaluation              grams.                                studies,   to award a follow-o"      contract    to the same con-
                                                                                                                             tractor   for analysis     of a.11 the Community Action Program
                                                                                                                             data that had been collected.
     J     cost plus            4- 9-69          104,452        Operations              Feasibility       of as-             The final report       was considered      satisfactory       by OEO offi-
           fixed  fee              to                                                   sessing     the impact               cials     and was used for examination          and correction     of man-
                                8-15-69                                                 of training       and                agement problems       in the OEO Training         and Technical    Assis-
                                                                                        technical      assistance            tance Division.        A follow-on    contract       was proposed    to de-
                                                                                        in Community Action                   sign and field     test a system for measuring            the impact of
                                                                                        Programs.                             training    in the OEO regions.
     K     cost plus            3- l-69          100,350        Volunteers              Evaluation        of VISTA           The reports      were considered        useful  by OEO; however,     in
           fixed  fee              to                           in Service              programs.                            our opinion,       their    objectivity     was questionable    in that
                                Z-28-70                         to America                                                   the contractor        hired    for the evaluation      had other large
                                                                                                                             contract    interests       in the VISTA program.
     L     cost    plus         6-24-69          127,812        Planning,               Report       on studies         of   The final    reports    were considered     satisfactory       by OEO and
           fixed     fee                                        Research        and     Upward       Bound pro-              were furnished       to the Deparunent    of Health,       Education,
                                2-z-70                          Evaluation              gram.                                and Welfare     which currently     is responsible       for the Upward
                                                                                                                             Bound program.
     M     Fixed    price      6-27-69            26,293        Program        Devel-   Assessment  of Commu-                The contract         was awarded,     in pert,      to help OEO develop     an
                                  to                            opment                  nity Action Program                  Internal      utilization       system for its evaluation         and study
                              11-15-69                                                  evaluations   and de-                information.           OEO considered      the contract    useful   in that
                                                                                        sign for dissemina-                  the contractor          pointed    at   system deficiencies.        The con-
                                                                                        tion of information.                 tractor      recommended contract          extension,    and a follow-on
                                                                                                                             contract      was awarded.
     N     Cost p1xs            6-27-69          151,163        Progran        Devel-   Evaluation  of the                   Tne final    report    draft    was revised      at no cost to OEO.
           flxed price                                          0pSEIlt                 Community Action                     The revised     report    was critiqued        by OEO officials       as con-
                                2-::-70       $3  163 433
                                               ==!zc=zL                                 Program planning    de-              taining   too much description           and not enough evaluation
                                                                                        velopment  program.                  and, hence, as of limited           usefulness.      The co"tracfar'*
                                                                                                                             overall   performance        was stated    as less than satisfactory.
?n most cases      these figures     represent          estimates        as shown I"     the contracts,         many of which         were    still     pending    final   audit   and settlement     at
 the completion      of our fieldwork.
                                                         APPENDIX III




                  DISCUSSED IN THIS REPORT

                                               Tenure of office
                                               From            To

    Phillip  V. Sanchez                 Sept.     1971     Present
    Frank C. Carlucci                   Dee,      1970     Sept. 1971
    Donald Rumsfeld                     %Y        1969     Dec. 1970
    Bertrand M. Harding     (acting)    Mar,      1968     &Y      1969
    R. Sargent Shriver                  Oct.      1964     Mar, 1968

   Wesley L. Hj ornevik                 Oct.      1969     Present
   Robert Perrin (acting)               Mar.      1968     Oct.    1969
    Bertrand M, Harding                 June      1966     Mar. 1968

  TRATION (note a>:
    Ernest Russell (acting>             Apr.      1971     Present
    Robert C. Cassidy                   Sept,     1967     Apr.    1971

    John 0. Wilson                      Oct.      1969     Present
    Richard Ottman (acting)             Jan,      1969     Sept. 1969
    Robert A. Levine                    Nov.      1966     Jan.    1969

    Louis Churchville   (Associate)     June      1970     Present
    Richard D, Peacock (acting)         Feb.      1970     June 1970
    Arch M&inlay                        Aug.      1969     Jan.    1970
    Kenneth Sparks (acting)             June      1968     Aug. 1969
    Herbert J. Kramer                   Jan.      1966     June 1968


                                           Tenure of office
                                           ---             -To
    Carol Khosrovi                    July    1971    Present
    Alfred Taylor (acting)            June    1971    July    1971
    Joseph P. Maldonado               Aug.    1970    June    1971
    Marvin J. Feldman                 Jan.    1970    Aug. 1970
    Robert Perrin    (acting)         Sept.   1969    Jan,    1970
    Theodore M. Berry (Assistant
       Director  for Community Ac-
       tion Program) (note c)         Apr.    1965    Sept.      1969

    H. Rodger Betts (acting)          Sept.   1971    Present
    Phillip    V. Sanchez             Feb.    1971    Sept. 1971
    Donald I. Wortman (acting)        Dec.    1970    Feb. 1971
    Frank C. Carluoci                 Dec.    1969    Dee, 1970
    William Bozman (acting>           Oct.    1969    Dec. 1969
    Theodore M, Berry (Assistant
      Director    for Community Ac-
      tion Program (note c>           Apr.    1965    Sept.      1969

    Carl A. Smith (acting)            MY      1971    Present
    Thomas E, Bryant (Associate)      Sept.   1969    Apr.    1971
    Thomas E. Bryant (acting>         Jan.    1969    Sept. 1969
    Joseph T. English                 Mar.    1968    Jan.    1969

  IN SERVICE TO     I@4 (note d):
    Carol Khosrovi                    Sept.   1970    July       1971
    Chester R. Lane (a.eting)         Jm.     1970    Sept.      1970
    Padraic Kennedy (acting>          July    31968   Jan.       1970
    Bill Crook                        Mar.    1967    June       1968

                                                          APPENDIX III

aPrior to June 1968 this     office    was called   the Office   of
b Prior  to 0EO"s September 1969 reorganization, this office
 was called the Office of Research, Plans, Programs and

'In September 1969 this position        was terminated   as an or-
  ganizational   entity and-responsibility      for the programs
  was shifted  to two newly created offices,        the Office of
  Program Development and the Office of Operations,

dp ursuant to Reorganization Plan No. 1 of 1971, on July 1,
 1971, the VISTA program was transferred  from OEO to ACTION,
 a new Federal agency.

U.S GAO, Wash., D.C.              31
                                               ,\     ,-,   ,   -             _-   - .. =,.:     d,,. .,
,,   --r   ..   -
                      -,                                                                       ! ‘L

                Copies  of this report are available     from the
                U. S. General   Accounting   Office, Room 6417,
                441 G Street, N W., Washington,     D.C., 20548.

                Copies are provided        without       charge to Mem-
                bers of Congress,        congressiona         I committee
                staff members, Government            officia Is, members
                of the press, college        libraries,      faculty   mem-
                bers and students.        The price to the general
                public   is $1 .OO a copy.       Orders     should   be ac-
                companied     by cash or check.