oversight

Need for Improving the Administration of Study and Evaluation Contracts

Published by the Government Accountability Office on 1971-08-16.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

Need For Improving The
Administration Of Study And
Evaluation Contracts                    Eqr64031(1J




Office of Education
Department    of Health,   Education,
  and Welfare




BY THE COMPTROLLER GENERAL
OF THE UNITED STATES
                        COMPTROLLER     GENERAL     OF      THE   UNITED   STATES
                                      WASHINGTON.    D.C.     20548




     B- 164031( 1)




 /   To the President   of the Senate and the
‘L
     Speaker  of the House of Representatives

                 This  is our report   on the need for improving    the admin-
     istration       of study and evaluation    contracts by the Office  of Ed-       - *.
r_                                                                                      :
                                                                                     2. c-
     ucation,       Department    of Health,  Education,  and Welfare.

              Our review  was made pursuant   to the Budget and Ac-
     counting    Act, 1921 (31 U.S.C. 53), and the Accounting  and
     Auditing    Act of 1950 (31 U.S.C. 67).

             Copies of this report    are being sent to the Director,
     Office of Management       and Budget; the Secretary    of Health,
     Education,    and Welfare;    and the Commissioner    of Education.




                                                    Comptroller            General
                                                    of the United          States




                             50 TH ANNIVERSARY                1921- 1971
        I     .
                   .       *.
        ’ ,
        ’     -.
        I.             I
                           CO&TROLLER GENERAL'S                            NEED FOR IMPROVING THE ADMINISTRATION
        I                  REPORT TO THE CONGRESS                          OF STUDY AND EVALUATION CONTRACTS
        I
                                                                           Office  of Education,   Department of
                                                                           Health,  Education,   and Welfare
                                                                           B-164031(1)


        I                  DIGEST
                           ------


                           WHY THE REVIEW WASMADE

                                   The Office    of Education  has entered into contracts      for -. studies.__
                                   and evaluations    of Federal educational   progT-‘-6"-aetermine
                                   whether these programs are meeting their       objectives.       The infor-
                                   mation obtained    is used in the development,     design,    and manage-
                                   ment of the programs and to inform educators        about the programs.

                                   The studies   are performed by public           or private       agencies,   organiza-
                                   tions,  groups, or individuals.

                                   The General Accounting        Office (G&)-reviewed      the Office      of Educa-
                                   tion's   ad-ministration    of these contracts      to determine     whether its
                                   policies   and procedures      were adequate for ensuring        that the infor-
                                   mation obtained       was useful and provided     the benefits      intended.

                                   GAO identified     86 study and evaluation       contracts,      totaling    about
                                   $22 million,     which had completion     dates after       January 1, 1969.       Of
                                   these contracts,      GAO selected   24 for review.         At the time of GAO's
                                   review,   14 of the contracts      had been completed at a cost of
                                   $2.2 million.      The remaining    10 contracts     were still       in progress
                                   and were estimated      to cost about $9.1 million.


    I                      FINDINGS AND CONCLUSIONS

                                   Office   of Education officials       considered     the information      produced
                                   by five of the 14 completed        studies    to be of limited       use.    The
                                   cost of the five studies       ($935,000)     represented     41.6 percent of
                                   the total   cost of the 14 completed contracts.

                                   The results   of the nine other completed contracts   were considered
                                   by the Office   of Education  to be adequate and useful.

                                   Two of the 10 ongoing         studies     may also   fall    short   of meeting   their
                                   objectives.

                                         --A study,    costing    $542,000,   to evaluate    curriculum   for the en-
                                           vironmentally      deprived   child probably     will   not meet the ob-
                                           jectives    set by the Office      of Education.



                                                                     1
                            Tear Sheet
                                                                                               AUG.1&1971
I
                                                                                                .       -            I
                                                                                                    .
                                                                                                                 .   I
                                                                                                            .
                                                                                                                     I


  --A $7 million      contract   to study the Follow Through program may .                          *       ,,*:'i
     not produce the information          desired unless the Office of Edu-                                          I
     cation clarifies       the objectives      of the study. (See p. 6.)                                            1

Numerous problems     were encountered      with     some of the studies.

  --Certain   of the studies lacked       sufficient         research,    test   data,
     and analyses to support their        conclusions.

  --One contained   little    or no original         data;    another    contained        in-
     accurate data.      (See p. 8.)

Weaknesses    in the administration     of the contracts  contributed                to
the failure    of these studies     to produce the desired results.

In a number of instances,          the contractors'    descriptions        of work to
be performed were not specific          enough to ensure that the           work per-
formed would provide the Office of Education with useful                     informa-
tion.     Written    agreements were not obtained on significant                changes
in the work.        Also contracts    were not monitored       closely     enough to
keep responsible         Office of Education officials       informed      on a con-
tractor's      progress.

Under such circumstances    it is difficult    to hold            the contractors
responsible  for poor performance.       (See p. 14.)

The following     two examples illustrate          the poor administrative
practices    of the Office of Education.

  --Office     of Education officials     concluded that research con-
     ducted under a $103,000 contract         was performed poorly and that
     the final    report    contained many unsubstantiated   statements.
     GAO attributes      the poor results   to a lack of specific   contract
     objectives.

  --Office   of Education officials     concluded that the report        pro-
     duced under a $200,000 contract      lacked sufficient        data or re-
     search findings    to be of value.     GAO believes    that this re-
     suited because the Office of Education did not formally             amend
     the contract    to include work it considered       important.     (See
     p. 19.)

If the Office of Education is to receive the benefits    intended
from study and evaluation   contracts,  improvements are needed in
the administration  of these contracts.

At the close of GAO's review,     the Department of Health,     Education,
and Welfare (HEW) was preparing      a guide for its project    monitors,
which was to deal with many of the problem areas discussed in
this report.   GAO believes   that the guide can result      in a signif-
icant improvement.



                                 2
           ’        .

     .         -,

 .       RECQ~NDATIONS               OR SUGGESTIONS
- .'
                        In preparing  the guide for project       monitors,   the Secretary    of HEW
                        should provide for inclusion   of:

                          --Guidance    to help ensure that the objectives     and requirements
                             of contracts   are clearly  understood by the contractors       and
                             that the scope of work is described    in sufficient     detail   in
                             the contract.

                          --The methods to be used in monitoring    the contractor's      progress
                             such as required site visits at specified   points     in the con-
                             tract period.

                          --Criteria    for use by agency personnel       in evaluating a final     re-
                             port prior    to acceptance   to determine    whether the contract     ob-
                             jectives   have been satisfied.

                          --The steps required  to be taken when considering   action    against
                             a contractor for poorly performed work, including    consultation
                             with legal counsel and contracting officials.

                        In addition,    the Secretary   should provide for the establishment         of
                        an orientation     course to acquaint  agency program personnel       involved
                        in the administration     of study and evaluation     contracts   with the
                        requirements    of Federal Procurement    Regulations   and agency
                                                                                        -  - instruc-
                        tions.     (See p. 28.)


         AGENCY ACTIONS AND UNRESOLVEDISSUES

                        The Assistant   Secretary,   Comptroller,     of HEW concurred     in GAO's
                        recommendations   and described    actions     taken or planned to imple-
                        ment the specific    recommendations      or to otherwise    improve contract
                        management in the Office     of Education.        (See p. 29.)


         I'MTTERS FOR CONSIDERATION BY THE CONGRESS

                        The use of contractors       by the Office   of Education    to conduct stud-
                        ies and evaluations       has increased   substantially   over the years.
                        This report   illustrates     the need for improved contract      administra-
                        tion so that the Office       of Education can realize     the maximum ben-
                        efits  from the funds expended for these studies.




         Tear Sheet
    .    - .

.   .’

                                      Contents
                                                                            Page

          DIGEST                                                              1

          CHAPTER

                1       INTRODUCTION                                          4

                    2   REPORTSWEREOF LIMITED USEFULNESSAND
                        OTHERCONTRACTWORKWAS NOT PROGRESSING
                        SATISFACTORILY                                        6
                            Completed contracts                               8
                            Ongoing contracts                                 9
                            Disposition  of contract results                 11

                        NEED FOR IMPROVEDPOLICIES AND PROCEDURES
                        IN ADMINISTERING STUDYAND EVALUATION CON-
                        TRACTS                                               13
                            Need for more specific    description      of
                              work to be performed                           14
                            Need for formal record of agreements
                              with contractor                                19
                            Need for effective   monitoring     proce-
                              dures                                          21

                        CONCLUSIONS,RECOMMENDATIONS, AND AGENCY
                        COMMENTS                                             28
                           Conclusions                                       28
                           Recommendations to the Secretary of
                             HEW                                             28
                           Agency comments                                   29

                        SCOPEOF REVIEW                                       31

          APPENDIX
                I       Letter dated June 1, 1971, from the Assis-
                          tant Secretary,  Comptroller, Department of
                          Health, Education and Welfare, to the Gen-
                          eral Accounting Office                             35

               II       Schedule of contracts    selected   by GAO for
                          review                                             40
                                                                        Page

      APPENDIX

       III       Principal  officials  of the Department of
                    Health, Education, and Welfare responsible
                    for administration  of activities  discussed
                    in this report                                       44

                                  ABBREVIATIONS

      GAO        General Accounting     Office

      HEW        Department    of Health,   Education,    and Welfare

      ERIC       Educational    Resources   Information    Center




.,-
    .
*       .’   ChPTROLLER GENERAL'S                       NEED FOR IMPROVING THE ADMINISTRATION
             REPORT TO THE CONGRESS                     OF STUDY AND EVALUATION CONTRACTS
                                                        Office of Education,   Department of
                                                        Health,   Education, and Welfare
                                                        B-164031(1)


             DIGEST
             ------


             WHY Th!E REVIEW WASM4DE

                  The Office   of Education has entered into contracts      for studies
                  and evaluations   of Federal educational  programs to determine
                  whether these programs are meeting their      objectives.     The infor-
                  mation obtained   is used in the development,     design,   and manage-
                  ment of the programs and to inform educators       about the programs.

                   The studies   are performed    by public     or private       agencies,   organiza-
                   tions,  groups, or individuals.

                  The General Accounting        Office (GAO) reviewed the Office of Educa-
                  tion's   administration     of these contracts     to determine     whether its
                  policies   and procedures      were adequate for ensuring       that the infor-
                  mation obtained       was useful and provided    the benefits      intended.

                  GAO identified     86 study and evaluation       contracts,      totaling    about
                  $22 million,     which had completion     dates after       January 1, 1969.       Of
                  these contracts,      GAO selected   24 for review.         At the time of GAO's
                  review,   14 of the contracts      had been completed at a cost of
                  $2.2 million.      The remaining    10 contracts     were still       in progress
                  and were estimated      to cost about $9.1 million.


             FINDINGS AND CONCLUSIONS

                  Office   of Education officials      considered     the information     produced
                  by five of the 14 completed studies          to be of limited      use.    The
                  cost of the five studies       ($935,000)    represented    41.6 percent of
                  the total   cost of the 14 completed contracts.

                  The results   of the nine other completed contracts   were considered
                  by the Office   of Education  to be adequate and useful.

                  Two of the 10 ongoing       studies    may also   fall     short   of meeting   their
                  objectives.

                      --A study, costing      $542,000,    to evaluate   curriculum   for the en-
                         vironmentally    deprived   child   probably will     not meet the ob-
                         jectives    set by the Office     of Education.



                                                  1
  --A $7 million       contract   to study the Follow Through program may                       ,
     not produce the information          desired    unless the Office of Edu-
     cation  clarifies       the objectives     of the study.   (See p. 6.)

Numerous problems     were encountered      with     some of the studies.

  --Certain   of the studies lacked       sufficient         research,    test   data,
     and analyses   to support their      conclusions.

  --One contained    little    or no original        data;    another    contained        in-
     accurate  data.      (See p. 8.)

Weaknesses    in the administration of the contracts   contributed                   to
the failure    of these studies to produce the desired   results.

In a number of instances,          the contractors'    descriptions        of work to
be performed       were not specific    enough to ensure that the           work per-
formed would provide the Office of Education with useful                     informa-
tion.     Written    agreements were not obtained       on significant          changes
in the work.        Also contracts    were not monitored       closely     enough to
keep responsible         Office of Education officials       informed      on a con-
tractor's      progress.

Under such circumstances    it is difficult     to hold           the contractors
responsible  for poor performance.       (See p. 14.)

The following     two examples illustrate          the poor administrative
practices    of the Office of Education.

  --Office     of Education officials    concluded that research   con-
     ducted under a $103,000 contract        was performed poorly and that
     the final    report contained    many unsubstantiated  statements.
     GAO attributes     the poor results   to a lack of specific   contract
     objectives.

  --Office   of Education officials     concluded that the report         pro-
     duced under a $200,000 contract      lacked sufficient        data or re-
     search findings    to be of value.     GAO believes     that this re-
     sulted  because the Office of Education did not formally            amend
     the contract    to include work it considered       important.     (See
     p. 19.)

If the Office      of Education is to receive    the benefits            intended
from study and evaluation       contracts,  improvements    are          needed in
the administration      of these contracts.

At the close of GAO's review,      the Department of health,     Education,
and Welfare   (HEW) was preparing     a guide for its project    monitors,
which was to deal with many of the problem areas discussed           in
this report.    GAO believes   that the guide can result      in a signif-
icant  improvement.
           -        *
      .        -.

. .
*     .’

           RECOiWENDATIONS
                         OR SUGGESTIONS
                        In preparing the guide for project        monitors,   the Secretary    of HEW
                        should provide for inclusion  of:

                          --Guidance    to help ensure that the objectives     and requirements
                             of contracts   are clearly  understood by the contractors       and
                             that the scope of work is described    in sufficient     detail   in
                             the contract.

                          --The methods to be used in monitoring    the contractor's      progress
                             such as required site visits at specified   points     in the con-
                             tract period.

                          --Criteria    for use by agency personnel       in evaluating a final     re-
                             port prior    to acceptance   to determine    whether the contract     ob-
                             jectives   have been satisfied.

                          --The steps required  to be taken when considering   action    against
                             a contractor for poorly performed work, including    consultation
                             with legal counsel and contracting officials.

                        In addition,    the Secretary   should provide for the establishment      of
                        an orientation     course to acquaint  agency program personnel     involved
                        in the administration     of study and evaluation     contracts with the
                        requirements    of Federal Procurement    Regulations   and agency instruc-
                        tions.     (See p. 28.)


           AGENCYACTIONSAND UNRESOLVED
                                     ISSUES
                        The Assistant   Secretary,   Comptroller,      of HEW concurred    in GAO's
                        recommendations   and described    actions     taken or planned to imple-
                        ment the specific    recommendations      or to otherwise    improve contract
                        management in the Office     of Education.        (See p. 29.)


           MTTERS FOR CONSIDERATION
                                  BY TEE CONGRESS
                        The use of contractors       by the Office   of Education    to conduct stud-
                        ies and evaluations       has increased   substantially   over the years.
                        This report   illustrates     the need for improved contract      administra-
                        tion so that the Office       of Education    can realize  the maximum ben-
                        efits  from the funds expended for these studies.
                                                                          .        -
                                                                               .



                                CHAPTER1

                              INTRODUCTION

       The study and evaluation     of educational   programs fi-
nanced with Federal funds has been emphasized increasingly
by the Congress; the Office of Management and Budget; the
Department of Health, Education,       and Welfare; and the Of-
fice of Education.    The Congress has indicated        that some
of this work should be done under contracts         with profit-
making and nonprofitmaking      firms outside the Federal Govern-
ment. This is illustrated       by section 412 of the General Ed-
ucation Provisions   Act (20 U.S.C. 1231 a), which authorizes
the Office of Education to enter into contracts          with private
and public firms for the purpose of obtaining          objective  mea-
surements of the effectiveness       of educational    programs.

        Contracts are awarded by the Office of Education for a
variety    of products and services,      including     various types
of studies and evaluations.          Study contracts      are awarded
for research which generally         seeks to develop or test new
knowledge, concepts, instruments,         or techniques or for the
collection     of basic data and statistics       which describe such
factors as educational      activities,     characteristics,     and com-
position    of target groups.       Other contracts     are awarded for
evaluation     which is defined by the Office of Education as
the process of gathering      and analyzing      information    about
program effectiveness     and impact to assist in making deci-
sions about alternative      courses of action.

        We compiled a listing      of about 1,300 contracts          of all
types, totaling      approximately      $276 million,     which were open
during fiscal     year 1969. Because a central record which
identified    contracts     by type was not maintained,          we made a
determination     of the number of study and evaluation              con-
tracts.     We identified     86 contracts,     totaling     about $22 mil-
lion, which had completion dates after January 1, 1969, and
selected 24, totaling        about $11.3 million,        for review.      We
selected a variety        of contracts,    considering      contract   size,
type, and the awarding activity           within the Office of Educa-
tion.




                                     4
      .   .
*.
 ..
* :
            At the time of our selection, 14 of the contracts   had
      been completed at a cost of $2.2 million.    The remaining 10
      contracts  were still in progress and were estimated to cost
      about $9.1 million.




                                    5
                               CHAPTER2

                  REPORTSWEREOF LIMITED USEFULNESS

                       AND OTHERCONTRACTWORK

                WAS NOT PROGRESSINGSATISFACTORILY

       The Office of Education's      objective    in awarding study
and evaluation     contracts    was to obtain timely and objective
information    that could be used in the development, design,
and management of its programs or that was of a quality
that could be provided to the educational            community.    Our
review of Office of Education records and discussions              with
agency officials     showed, however, that the results          of five
of the 14 completed contracts         included in our review did
not fully meet this objective.         The cost of the five studies
 ($935,000) represented      41.6 percent of the total cost of
the 14 completed studies.         The results   of the other nine
completed contracts,       costing about $1.3 million,      were con-
sidered adequate by the Office of Education and were useful.
 (See p.lland     app. II for details     on use.)

         Although a final judgment had not been made by the Of-
 fice    of Education on the 10 ongoing studies--expected         to
cost     about $9.1 million--the       record indicates   that one of
the     studies estimated to cost about $542,000 probably will
not     meet the objectives     specified    by the Office of Educa-
tion.
       Information  on an ongoing $7 million   contract   indicated
that, unless the Office of Education gave immediate atten-
tion to specifying    what it expected to obtain from the con-
tract,    there was considerable  likelihood that the product
it received would not be as useful as desired.         In several
cases the files    did not contain for our consideration      up-
to-date reports on the work in progress.

      Because of the absence of a system at the Office of
Education for evaluating   and recording   the use of reports
produced under study and evaluation     contracts, we had to
rely on discussions   with Office of Education officials     and
available  records for assessment of report quality      and use.


                                    6
       - .
        - .
*.
 ..
 . :
               The five completed contracts    which produced results
        of limited usefulness     and the two ongoing contracts     with
        potential    problems are listed below followed by comments
        made by certain Office of Education officials       or consultants
        hired by the Office of Education.       Additional  details    are
        given in appendix II.      In chapter 3 we discuss the need for
        improving the policies     and procedures for administering
        study and evaluation     contracts  to provide greater assurance
        that such contracts    will produce useful information,
                                                       Approximate
                 Title      of study                      cost

        Completed:
           Establishment     of a National
              Planning Congress in the
              Field of Higher Education            $     200,000
           National Evaluation     of
              Project Follow Through                     146,100
           Analysis of the 1968 Survey
              of Compensatory Education                   80,579
            Study of Students Graduated
               from the Public Schools of
               (city deleted)                            103,172
           Design, Test, Operate and
              Evaluate Institute    Informa-
              tion System for National
              Defense Education Act--l966                404,880
                    Total                                            $     934,731

        Ongoing:
            Evaluation   of the Effective-
               ness of an Enriched
               Curriculum in Overcoming
               the Consequences of
               Environmental    Deprivation              542,180
            Longitudinal     Evaluation   of
               the National Follow Through
               Program                                 7,028,805
                    Total                                                7,570,985
                    Total                                            $8,505,716


                                         7
COMPLETEDCONTRACTS

    1. The Chief of the Planning, Evaluation,      and Reports
       Staff, Bureau of Higher Education,   criticized    the
       $200,000 study of higher education planning be-
       cause:

       YI'here does not seem to be much in the report
       by way of data or research findings.        *** In
       short, we have a negative report,      filled    with
       broad and frequently     obvious generalizations
       and opinions,    lacking in data and proposing
       Federal legislation     that contains little     or
       nothing original."
       This official  recommended limited   distribution be-
       cause, in his opinion,    there did not appear to be
       much of substantive   value in the report.

    2. A report produced under the $146,100 contract           for
       the evaluation     of Project Follow Through was criti-
       cized by Office of Education officials,        including
       the Director     of the Follow Through Branch.       These
       officials    stated that the report lacked significant
       test data and that it did not accurately        reflect
       the analyses that had been completed.         They also
       considered it to be of limited usefulness         because
       of significant     changes in program emphasis during
       the contract period.       For these reasons they felt
       that the report was misleading       and that distribut-
       ing it, without revision,       would have been irrespon-
       sible.     The Director of the Follow Through Branch,
       however, agreed to accept the report without revi-
       sion because he felt that there was nothing to be
       gained from further     negotiation   with the contrac-
       tor.

    3. In the case of the $80,579 analysis of the 1968
       survey of compensatory education,       the project moni-
       tor stated that the materials      delivered   by the con-
       tractor  were of variable  quality     but were, for the
       most part, unacceptably   poor.     Part of the analysis
       had to be redone by another contractor       at addi-
       tional cost to the Government.

                                8
    4. The project monitor for the $103,172 study of stu-
       dents graduated from a public school system stated
       that:
       "**Jr the research it describes was poorly con-
       ducted and the report contains many unsubstan-
       tiated statements which would be misleading   to
       readers."

       Another Office of Education      official concluded that
       the report was of absolutely      no use to the Office
       of Education.

    5. An important objective       of the $404,880 contract
       was not accomplished in that information           considered
       by the Office of Education to be needed for program
       planning and decisionmaking        purposes was late by as
       much as 1 year.        The contract was for designing,
       testing,    operating,    and evaluating    an information
       system for various institutes'         programs, such as
       the Arts and Humanities Institutes          Program and the
       Experienced Teacher Fellowship Program, which were
       established    in accordance with the National Defense
       Education Act of 1966.

       An important objective   of the contract was to de-
       velop timely statistical    data on persons applying
       for, and participating   in, the various institute
       programs.   The data on the participants    in the sum-
       mer of 1966 were to be submitted to the Office of
       Education in time to be used for planning the pro-
       gram for 1967. According to Office of Education
       officials,  however, correct data were not received
       in time to be of benefit    for this purpose; there-
       fore the value of the data was diminished.
ONGOINGCONTRACTS

    1. The Office of Education has recognized that one of
       the major objectives   of the $542,180 contract      for
       the evaluation   of the effectiveness     of an enriched
       curriculum   for the environmentally    deprived child
       probably will not be achieved fully,       because the
       unique aspects of the curriculum      which affect the

                                9
                                                                   .   .


   child's   performance will not be identified    fully.  '               ' '
   The Office of Education became aware of this prob-
   lem in 1967 at which time the project monitor
   stated that "no valid conclusions     about the effec-
   tiveness of the 'enriched'   curriculum    will be pos-
   sible."

   Furthermore,    on October 12, 1970, the contractor
   informed the    Office of Education that all of these
   aspects would    not be identified   in the final report
   and that one    must speculate on some of the results
   and interpret    as well as possible   from the data.

   In addition,   the curriculum    being evaluated was de-
   veloped by the same contractor.        Although the ef-
   fect of the contractor's     vested interest    on the
   objectivity   of his evaluation     cannot be determined,
   it appears desirable     to avoid situations    where the
   contractor   evaluates the effectiveness      of work that
   he has performed.
2. Action had not been taken to clarify        objectives     on
   the $7 million     study of the national    Follow Through
   program to be completed in December 1971, even
   though independent consultants      and an Office of Ed-
   ucationofficialhad      concluded that the objectives
   of the study had not been specified       in sufficient
   detail   to ensure a successful    outcome.     As a result
   of our review, Office of Education officials           agreed
   to develop a specific     work statement.     They also
   informed us that they planned to establish          a team
   of experts to monitor the study for the remaining
   months.




                           10
*.        1


 .   ‘.

               DISPOSITION OF CONTRAm RESTJLTS
              : -
               Contracts of limited usefulness
                      The $200,000 study of higher educationplanning          oventu-
               ally was placed in the Office of Education's          Educational     Re-
               sources Information       Center system.    The Office of Education,
               however, did not reproduce or disseminate         the report by
               other means, such as distribution        through the Government
               Printing    Office, as is done with studies deemed fully          satis-
               factory.      The results   of the $146,100 evaluation    of a Fol-
               low Through project and the $103,172 study of students grad-
               uated from a public school system were withheld from the
               Educational     Resources Information    Center and were never dis-
               tributed    formally by other means, According to Office of
               Education officials,       only limited use was made of the
               $80,579, 1968 survey of compensatory education and much of
               the work had to be redone by another contractor.
                       The Office of Education considered the final report of
               the $404,880 study contract       for designing,  testing,   operat-
               ing, and evaluating      an information   system for various in-
               stitute    programs to be useful,     even though the interim prod-
               uct had not been received in time to be useful in planning
               program activities     for the summer of 1967. Information        has
               been extracted     from the final report on several occasions
               for use in Office of Education publications         and for satisfy-
               ing public requests.

               Contracts     considered   useful

                       The results   of the remaining nine completed studies
               were considered acceptable and useful to Office of Education
               needs,     For example, the Bureau of Education for the Handi-
               capped within the Office of Education intends to utilize
               the results     of the $430,487 study of the need for educational
               manpower for handicapped children      and youth in its planning
               efforts    with State education agency officials    to help them
               to more adequately assess manpower needs in special educa-
               tion.     Also part of the information    will be used in conduct-
               ing a future Bureau program,

                        Another illustration of a useful end product was the
               report     produced under the $24,750 study of special adult
                                                               .    .


                                                                        , .
basic education projects.     The Office of Education's     proj-
ect monitor informed us that the product had been dissemi-
nated to State officials    as well as to officials   within the
Office of Education.     For additional  information  on the
usefulness  of reports produced under these nine contracts,
see appendix II, page 41.




                                12
            .    .


I.
  . .
 .      :

                                              CHAPTER3

                           NEED FOR IMPROVEDPOLICIES AND PROCEDURES

                       IN ADMINISTERING STUDYAND EVALUATIONCONTRACTS

                       Because of numerous new educational          programs enacted by
                the Congress in recent years in such areas as research and
                training,   the administrative     responsibilities        of the Office
                of Education have increased sharply.            To assist the Office
                of Education in carrying       out the provisions       of these many
                programs, legislation,      such as section 412 of the General
                Education Provisions     Act, authorizes      the Commissioner of Ed-
                ucation to enter into contracts        with public or private       agen-
                cies, organizations,     groups, or individuals         for data-
                gathering   studies and evaluations.
                        In some respects this has created new problems for Of-
                fice of Education program and contracting        personnel who face
                a number of problems that normally would not be present in
                the procurement of such specifiable        items as supplies and
                furniture.     In this latter   type of procurement,    the product
                to be produced or supplied under the contract         can be de-
                scribed with enough exactness to permit the buyer to place
                considerable    reliance  on the description    of the item to be
                procured or the work to be done to ensure that he will re-
                ceive a satisfactory     end product.

                       On contracts  for studies and evaluations    of complex ed-
                ucational   programs, however, it is difficult    to specify
                exactly what is to be accomplished and what is to be submit-
                ted as a final product,      The Office of Education expects a,
                product under this type of contract,     which is usually in
                the,form of a report,    but the methodology to be used in con-
                ducting the study and the nature of the final report is not
                always subject to exact definition,
                       Although Federal Procurement Regulations     and agency in-
                structions    provide considerable  guidance on contracting    in
                general, we believe that the problems revealed by our review
                point out the need for more specific      direction  as to how
                Office of Education personnel are to deal with the relatively
                unique problems associated with the management of study and
                evaluation    contracts.   We noted administrative   weaknesses

                                                  13
                                                                           .   .


in 20 of the 24 contracts    reviewed.    We believe that, to * '                  .-'I
effectively  cope with such problems, the following      improve-
ments are required   in contract   administration.

      1. More specific    descriptions     of work to be performed.
      2. Formal records of agreements with the contractor.
      3. Effective   monitoring     procedures.

NEED FOR MQRE SPECIFIC DESCRIPTION
OF WORKTO BE PERFORMED

        If the Office of Education is to obtain useful and re-
liable    information      from studies performed under a contract,
it is essential        that there be a clear understanding           between
the contractor       and the Office of Education concerning the
work to be performed.          According to the Federal Procurement
Regulations,      the contract      specifications       should describe
clearly     and accurately     the technical       requirements   for a
material,     product,     or service,     including    the procedure by
which it will be determined.that              the requirements    have been
met. The need for clear work statements also is recognized
in procurement regulations           issued by WEW, which are to be
followed by the Office of Education.

      In each of the contracts   reviewed, the study was re-
quired to be made in accordance with a proposal submitted
by the contractor   to the Office of Education.    In a number
of instances the contractor's    proposal was not specific
enough to ensure that the work to be performed would pro-
vide the Office of Education with useful information.

      For example, after the Office of Education had deter-
mined that the final report submitted by the contractor
under the $103,172 contract      for the study of students gradu-
ated from a public school system was of poor quality,       a
meeting was held between Office of Education officials        and
members of the Office of General Counsel, HEW, to discuss
the possibility   of withholding     final payment.  A memorandum
summarizing the results    of this meeting stated that:
      “A    subsequent review of the description     of activ-
      ities    contained in the contractors    proposal,  which
      is incorporated     into the contract,   revealed that
      the activities     are not specified   in adequate


                                   14
      detail to form a basis for the government to
      claim non-performance,     The government's  task of
      demonstrating   that 'best efforts'   were not devoted
      would be nearly impossible."

For the above reason, a decision was made to accept the re-
port "despite its serious deficiencies," and final payment
was made.

       In another case the Office of Education did not clearly
specify what it expected the contractor           to do under the
$404,880 study contract      for designing,     testing,   operating,
and evaluating    an information   system for various institutes'
programs.     The Office of Education intended that the contrac-
tor develop timely statistical       data, which were to be used
for program planning and decisionmaking           purposes, on persons
applying for, and participating       in, these institutes,          The
requirement    for submission of interim data before the con-
tract completion date, however, was not incorporated              into
the contract.      The contractor  forwarded the information           to
the Office of Education at least 13 months after it was ex-
pected, which was too late to be of any use for the subse-
quent year's planning for institute         participation.

      In a letter dated December 27, 1966, the contractor   ac-
knowledged its tardiness  in supplying statistical  data to
the Office of Education and stated that the

      'I*** best solution       on your [the Office of Educa-
      tion's]   part would be to set down complete speci-
      fications    including     timetable  and see that we
      [the contractor]       attach our signature."
      In October 1967 an official    of the Division   of Educa-
tional Personnel Training,    Office of Education,   stated in a
memorandum to the Contracts Division      of the Bureau of Ele-
mentary and Secondary Education that:
      "Due largely to the fact that this contract          is
      the first   of its type in which this Division has
      been involved,       it has been found that some of the
      highly technical        specifications were not suffi-
      ciently   definitive      and explicit to permit the con-
      tractor   to complete all of the work contracted        for
      within the original        time span."

                                   15
                                                                    .   .


        We were informed by the Off ice of Education's    project  :*:
monitor that the contractor's      work load had been reduced in
the latter    years of performance of the contract     so that the
contractor    would be in a position   to complete the required
statistical    tables more timely.    These tables were consid-
ered useful to the Office of Education.

       The $7 million      study of the national      Follow Through
program is another example of the need for more specific
statements of contract objectives           and requirements.      Accord-
ing to Office of Education officials,           it was recognized from
the beginning     of the program that precise specifications
for the evaluation       task could not be provided in advance.
In fact, the project monitor informed us that one of the
contract   objectives     was to have the contractor        develop the
detailed   specifications       of the evaluative    study.    The im-
portance of clear objectives          on this contract,     however, was
recognized    in a letter,      dated July 3, 1969, to the Commis-
sioner of Education from the Assistant            Secretary for Plan-
ning and Evaluation,        HEW, in*which he stated that:
      I'*** an evaluation    of this magnitude because       of
      its extreme complexity rehires       the closest      pos-
      sible monitoring    and the performance of the        con-
      tractor   *** must be measurable against items         spe-
      cifically    called for in the contract."

        We noted that, during the period of contract perfor-
mance, the Office of Education had the study reviewed by
consultants     and in-house personnel on four occasions.      The
results    of each review indicated   that detailed specifications
had not been set forth.
        The initial   review of the interim report submitted by
the contractor      covering the period July 1968 to November
1969 was conducted by a consultant.        In his comments on the
report,    the consultant    stated that:
     "'The major weakness of the program is, as admit-
     ted in the report itself,       its inconclusiveness
     from an evaluation     standpoint.     It would seem
     that T'Follow-Through11 would have certain        identi-
     fiable  objectives   for each year of operation and
     that some evaluation      of progress in achieving
     these could be made."

                                 16
      .     .

..
* :       In'addition, he concluded that "The absence of actual evalu-
          ation in an evaluative  study *** could be a major weakness."

                  In May 1970 a group of consultants          reviewed    the con-
          tract    and stated that:

                  "The major concern of this reviewing            'team' ***
                  lies in the design of the project         in terms of its
                  actually   evaluating    Follow-Through     Projects.
                  While the potential      of most information        collected
                  for this purpose (evaluation)        was explained,
                  there seems to be an absence of a definite              de-
                  tailed plan designed to serve the major objective
                  of evaluation.      In fact, there seems to be some
                  hesitancy   to evaluate-- in either a comparative
                  sense-- that is to compare project against project
                  or to evaluate against specific         goals."

                 Furthermore,  at our request an Office of Education of-
          ficial   and an outside consultant  reviewed the contractor's
          performance and informed us on November 4, 1970, that:
                  ‘I***  in the    absence of any detailed        statement of
                  work ***, it      will be impossible        to determine
                  whether the      contractor     is actually    doing the job
                  which he is      supposed to do. Apparently          several
                  millions    of   dollars    of effort has been contracted
                  for without      a specification     of the tasks and prod-
                  ucts of that      effort."

                We discussed these comments with officials          of the Fol-
          low Through program to determine what action they had taken
          to remedy the situation.       The Director     of the Follow Through
          Branch stated that, after the various criticisms           were re-
          ceived, he had talked to the contractor          about the matter.
          He stated, however, that he had not put any specific            objec-
          tive into writing.     He informed us that immediate priority
          would be given to correcting      this matter.       The Acting Direc-
          tor of Program Planning and Evaluation,          Bureau of Elementary
          and Secondary Education,     stated that the Office of Education
          also was considering     the possibility      of having a group of
          consultants   monitor the contract,       which would entail    some
          monitoring   at the contractor's      site in an attempt to rectify
          the current situation.


                                                 17
                                                                        .   .
                                                                                .


       We believe that, had more specific     work statements                   : .
been included in the two completed contracts         cited above,
the chances of the Office of Education's        receiving   a better
product would have been increased substantially.           As pre-
viously discussed,    agency officials   advised us during our
re-view that they planned to develop a specific         work state-
ment for the $7 million     contract and to establish      a team of
experts to monitor the contract.       We believe that the Office
of Education should follow up on the implementation          of these
actions.




                                 18
    NEED FOR FORMALRECORDOF
    AGREEMENTS WITH CONTRACTOR

           It is a well-established       policy within the Federal Gov-
    ernment that agreements reached by the contracting          agency
    with the contractor,     which change the scope of work, should
    be put in writing.      In some instances,      however, suggested
i   changes or improvements, essential          to the scope of work,
L
    were not agreed to, in writing,         by the contractor  and the
    Office of Education.        We believe that this contributed      to
    the failure   of the Office of Education to receive the qual-
    ity report that it expected and placed it in an awkward po-
    sition   in dealing with the contractor,

            The importance of written    agreements is illustrated    by
    the problems associated with the $200,000 study of planning
    in higher education.       This study was conducted in accordance
    with the contractor's      proposal which was reviewed formally
    within the Office of Education.        According to the reviewers,
    the proposal lacked sufficient       detail   to ensure that the con-
    tractor    understood the objectives     of the study; therefore   a
    meeting was held between the Office of Education and the con-
    tractor    to clear up certain matters prior to contract       award.
    According to a memorandum of the meeting, it was orally
    agreed to incorporate      several additions    to the scope of work
    in the contractor's     proposal.    Parts of the agreements were
    as follows:

         1. Committees to be involved in the project were to in-
            clude a mixture of research personnel,     teachers,
            students, and others, to present a balanced picture
            of higher education and to avoid overconcentration
            of those responsible   for administration.

         2. Products to be delivered  at the end of phase I of
            the project were to include a series of analytic   re-
            ports or statements of higher education issues, prob-
            lems, and needs,

            The Office of Education, however, never formally     re-
    quested a revised proposal from the contractor       to incorporate
    the changes discussed at the meeting.       Office of Education
    officials    informed us that, throughout   the contract   period,
    they were aware that the contractor      was not adhering to the


                                     19
agreements reached and, on at least one occasion, advised        *                 .-'
the contractor  of that fact.   We found no written   agreements                    ' '
prior or subsequent to contract award, however, which indi-                    .
cated that the contractor   was obligated by the changes dis-
cussed at the meeting or that the contractor     had agreed to
correct the deficiencies  brought to his attention.

       The criticisms     of the contractorls    interim and final re-
ports by officials       of the Office of Education and/or JXEW
were concerned primarily       with the failure     of the contractor
to complete the work orally agreed upon by the parties to
the contract      prior to award.    Several criticisms    of the in-
terim report,      which was submitted about 1 year after the con-
tract award, were that the contractor         did not seem to under-
stand the meaning of the project,         the study appeared to have
involved only a very select group of persons and institu-
tions,   and the report was considered,to        be high in rhetoric
but low in substance.

        The criticisms     of the final',report      were essentially    the
same. For example, one of the reviewers stated that the re-
port did not have much substantial            value.     Other reviewers
added that the report lacked a data base for recommendations
and did not contribute        in any way to information        about plan-
ning needs.        The project monitor informed us that the con-
tractor    had not included a comprehensive spectrum of discus-
sions with persons, including         students,      faculty members, and
members of the business community, who could have contributed
greatly to the project.

       The problems associated with the .$146,100 evaluation    of
the Follow Through program further     illustrate the importance
of having major changes in the scope of work agreed upon in
writing.    This contract was awarded in August 1967. Between
that time and about June 1968, the contractor     was primarily
gathering   data on 29 different  Follow Through projects.

       When the contractor      started to analyze the data col-
lected on the 29 projects,         as required in the contract,    the
Office of Education decided that an analysis,more          sophisti-
cated than originally      called for in August 1967 would pro-
duce a more meaningful report on the true impact of the na-
tional   Follow Through program.



                                    20
        The contractor   was told of this decision;     however, it
informed the Office of Education that it did not have the
capacity to perform the more sophisticated        analysis.     As a
result,    the Office of Education had another contractor        per-
form the analysis      but asked the original   contractor   to in-
clude the results      in its report.   We found no written     agree-
ments between the Office of Education and the two contractors
concerning this arrangement.
       The Director of the Follow Through Branch informed us
that the contract never had been amended formally to incor-
porate the additional    requests of the Office of Education.
The Director stated, however, that the contractor's     final
report included the additional    analyses but that the con-
clusions drawn from them by the contractor     had been criti-
cized by the Office of Education.
       According to the Director    of the Follow Through Branch,
the Office of Education accepted the report because it ful-
filled   the contract requirements,    as stated in 1967, which
never had been amended formally,       He stated, however, that
the report had not been released to the public because of
the belief    that it would be misleading.
      We believe that, had written     agreements been reached on
needed changes in the scope of work to be performed under
the two contracts    cited above, the chances of receiving    more
useful products would have been increased and the Office of
Education would have been in a better position      to hold the
contractors   responsible  for performing the intended objec-
tives of the contracts.
NEED FOR EFFECTIVE MONITORINGPROCEDURES
        Although the contractor      is responsible     for timely and
satisfactory      performance of its contract,       some form of mon-
itoring     is necessary to keep the Government informed of the
contractor's      progress and to identify      potential   problem
areas.      Monitoring   of the contract could consist of visits
to the contractorls       site, written    progress reports from the
contractor,      and timely actions to deal with problems dis-
closed during contract performance.



                                  21
Need for site visits  and                                                . -
adequate progress reports

       According to an HEWprocurement manual dated January 31,
1964, project monitors were responsible         for guiding the
technical    aspects of the projects     and supervising    the scope
of the contractorss      work, HEW's procurement manual, as well
as the Office     of Education's   contract and grant management
guidelines    dated June 1970, assigned the same responsibili-
ties to these persons,       We found no written guidelines,      how-
ever, for use by project monitors in determining          what they
should do to monitor contractors'        progress effectively.
Our discussions     with many of these monitors revealed that
the Office of Education       had not established    any detailed
guidance for their use.

      Site visits   were made on only 12 of the 24 contracts
from the time the contracts were awarded through September
1970; contract    periods ranged from about 1 to 6 years.    For
two of these contracts,    only one visit  was made on each,
although both contracts    had been outstanding  for over
5 years.

       Some files contained comments on the progress of the
work and others did not.    Most contracts required progress
reports to be submitted by the contractor.     In a number of
cases, however, the reports were too sketchy to keep the
project monitor informed as to the progress of the work.

     We believe that the failure    of the Office of Education
to provide its'project    monitors with detailed   guidance on
methods to be used in monitoring    contractors'   progress has
been a contributing    cause of the limited usefulness    of some
of the study results.

       For example, difficulties   have been encountered with
the $542,180 contract for the evaluation       of the effectiveness
of the enriched curriculum for the environmentally         deprived,
child.    The curriculum was developed under a contract with
the Office of Economic Opportunity,     and the Office of Educa-
tion contract   was awarded for the purpose of evaluating        this
curriculum.    The lack of effective   monitoring    appeared to be
a major cause of the difficulties     encountered with the of-
fice of Education contract,

                                 22
       The first  visit     to the contractor    site by Office of Ed-
ucation officials       to review the progress of the work was
made by a team of three reviewers and the Office of Educa-
tion project monitor in September 1967, more than 3 years
after the project       started.    Apparently   one reason for this
visit   was that officials       of the Office of Economic Opportu-
nity had discovered in an earlier          visit   that the contrac-
tor's work was not progressing         as satisfactorily    as expected.

      During the Office of Education visit,         it was discovered
that the contractor      had not achieved a major contract       objec-
tive which was to identify        the unique aspects of the curric-
ulum contributing     primarily     to the increased knowledge of
the participating     children.      We were informed by the former
Director of the Division        of Elementary and Secondary Educa-
tion Research that such identification          was important so that
the results     of the study could be disseminated      to other
schools.    He said that this was why the evaluation         had been
made. The failure      to achieve this objective      had not been
communicated to the Office of Education by other means, such
as progress reports,      prior to the 1967 visit.

       Primarily    because of the failure to achieve this major
objective,     the Office of Education project monitor,    in a
letter    to the Acting Chief of the Instructional    Materials
and Practices      Branch dated October 19, 1967, recommended
termination      of the contract by August 31, 1968, for the fol-
lowing reasons.

      1. It would not be possible to form valid conclusions
         about the effectiveness  of the enriched curriculum.

      2. It would not be possible  to say anything about vary-
         ing lengths of exposure to the curriculum.

      The Acting Chief of the Branch agreed with the recom-
mendation to terminate;    however, the contract was continued.
We found no documentation     supporting the decision to con-
tinue the contract.     The former Director  of the Division    of
Elementary and Secondary Education Research informed us,
however, that he had decided to continue     funding  the project
for the following   reasons.



                                  23
                                                                         .   .
                                                                                           .
                                                                                      .:
                                                                                     . .
      1. A significant   amount of money had already        been in-
         vested in not only the enriched curriculum          but also            l




         in its evaluation.

      2. The enriched    curriculum    was important   and needed to
         be evaluated.

      3, Two reviewers had not recommended termination but
         had suggested that there be an improvement in data-
         gathering  procedures.

       In our opinion,    the critical   problem with this contract
was that Office of Education officials         were not aware that
the contractor     was having problems until     the September 1967
site visit,    more than 3 years after the work started,        The
former Director     of the Division    of Elementary and Secondary
Education Research informed us further         on October 2, 1970,
that, had he been aware of the problems earlier          in the con-
tract period,    it possibly could have made a difference       in
his decision to continue funding.         He stated that action
could have been taken to correct the problems and that, if
that had failed,      the contract could have been terminated
since the investment in the project would not have been as
great as it was in September 1967. He added that by Septem-
ber 1967 it had been too late to change the direction          of the
contractor's    efforts,

Need to initiate    timely actions
to correct identified     problems

       An important part of a project monitor's        responsibili-
ties is to deal effectively      with problems identified       during
the contract    period.    Such problems may be detected through
the use of site visits      and progress reports or through the
assessment of interim and final products submitted by the
contractor,     Assessment of the final contract product is
required by Federal Procurement Regulations         before accep-
tance of the product by the Government.         Within the Office
of Education this assessment helps the project monitor dis-
charge his responsibilities      for seeing that the product is
responsive   to the intent and purpose of the study.          As
stated in Federal Procurement Regulations,        assessment pro-
vides the agency with a record of contractors'          past perfor-
mances for use in awarding new contracts.

                                  24
                                             .



       .
.. *
. .
                  We did not find any agencywide criteria     for use by a
           project monitor in determining     whether the contractor's      pro-
           duct was, in fact, responsive.      Furthermore we did not find
           any guidance for the monitor‘s     use in dealing with a contrac-
           tor whose performance was unacceptable,       nor did we find a
           central location    where Office of Education officials     could
           go to determine the quality     of a contractor's    past perfor-
           mance. As a result,      there seemed to be a lack of understand-
           ing within the Office of Education concerning what could and
           should be done when a contractor     performed poorly, which
           caused Office of Education officials      to deal inconsistently
           and ineffectively    with problem contractors.

                 For example, an official       stated with reference   to the
           $146,100 evaluation       of the Follow Through program that he
           was not familiar     with procedures for terminating      a contract.
           He stated that he had been "pulled off a college campus" to
           head the program but never had been given any training           in
           contract   administration      by the Office of Education.    A  for-
           mer project monitor on the $103,172 study of students grad-
           uated from a p,ublic school system stated that he had not at-
           tempted to terminate the contract because his supervisors
           had considered it too much trouble.

                   When an attempt was made to hold the contractor      respon-
           sible for its work, no set procedure was followed,          For ex-
           ample, the report produced under the $103,172 study was re-
           viewed by various persons within the Office of Education,
           including    the project monitor and project    co-monitor.     The
           project monitor inquired whether payments on the contract
           could be withheld but was told by an official        of the Office
           of General Counsel, HEW, that, primarily      because of the
           vagueness of the contract work statement,       it would be very
           difficult    to hold the contractor responsible     for poor con-
           tract performance.

                We discussed this matter with the official      of the Office
           of General Counsel, and he informed us that he felt that ac-
           tion against the contractor    would have been difficult      be-
           cause the contract work statement was vague and at the time
           there was some disagreement between the project monitor and
           the project  co-monitor  concerning the seriousness      of the



                                             25
                                                                  . _..

                                                                           .:
                                                                          . .
deficiencies  of the report.1  He stated that, if a formal
review of the report by a panel of experts had supported the *
deficiencies,  he would have been in a much better position
to take action against the contractor.

       In the case of the $80,579 analysis of the 1968 survey
of compensatory education,    a systematic approach was not
taken to hold the contractor    responsible  even though Office
of Education officials   decided to have part of the work re-
done because it had been performed so poorly.       The misunder-
standings and lack of coordinated     action on this contract
are discussed below.

       During the period of contract     performance,  the Office
of Education became aware that the work had not progressed
as planned and, on three occasions, had assigned persons
to assist the contractor     in developing the statistical     tables
required in the work statement.       The Assistant   Director   of
Program Planning and Evaluation,      Bureau of Elementary and
Secondary Education,    informed us that the tables completed
as a result of this assist work were the only ones accept-
able to the Office of Education.
       Several Office of Education officials       made comments on
the poor quality      of the work, and on February 25, 1969, the
Director   of Program Planning and Evaluation       wrote to the
contractor    and stated that "most of the important tables
were incorrect     and unusable."    In July 1969 the contractor
submitted a voucher for the total amount of the contract.
The voucher was not paid at that time, and in November 1969,
over 3 months later,       the Chief of Evaluation   Design, Program
Planning and Evaluation,       Bureau of Elementary and Secondary
Education,    requested the Fiscal Branch to withhold payment.
      In March 1970 the Assistant     Director  of Program Plan-
ning and Evaluation   authorized   payment and stated that the
Office of Education had no legal way to continue withholding
payment even though there had been dissatisfaction        with the
final report and the quality     of the contractor's    work.  The
payment was made on April 20, 1970.


1 In our discussions   with the project monitor and the project
 co-monitor,   we found that they later agreed that the report
 had serious deficiencies.
                                26
         ..


.   .
                  Our discussions  with this official    revealed that he had
        .not consulted with legal counsel concerning the legality           of
         withholding      payment.  He informed us that, when he autho-
          rized final payment in March 1970, he was under the impres-
          sion that all but about $20,000 of the total $80,579 had
          been paid to the contractor,        He stated that, had he known
          that none of the $80,579 had been paid at that time, he
         would not have authorized       the payment.     In a memorandufn
          dated June 15, 1971, the Office of General Counsel, HEW,
          stated that no practicable       remedial action against the con-
          tractor    appeared to be available     to the Office of Education.

               As discussed above, the contract-monitoring         function
        has not been carried out sufficiently       well to keep Office
        of Education officials     informed regarding the progress of
        this work so that timely decisions      can be made. Also the
        Office of Education has been placed in an unfavorable             posi-
        tion in dealing with contractors      who perform poor quality
        work, because there is no systematic       approach for assessing
        the end product and taking follow-up       action against the con-
        tractor.     Since records of contractors'     performances are
        not maintained in a central location,       essential    information
        is not available    for persons making decisions      on new con-
        tract awards.

              We believe that monitoring    problems have occurred
        partly because of the complex and subjective        nature of some
        of the studies conducted under contract.        IJnder a contract
        it is reasonable to expect some disagreements        between per-
        sons within the Office of Education and between Office of
        Education personnel and contractors      concerning the progress
        being made and the quality    of the work,

               In our opinion,     the use of panels of experts to assist
        a project monitor in carrying        out his responsibilities     would
        eliminate    some of the disagreements      and would place the Of-
        fice of Education in a stronger position          to deal effectively
        with contractors      that encounter problems in completing their
        work.     Such panels could be used to evaluate periodically
        the progress of relatively       large contracts.      This was done
        on the contract     for evaluation    of the effectiveness    of the
        enriched curriculum.



                                          27
                                                                            .        .
                                                                                I-




                                                                                                 -.
                                                                                             ,


                                                                                         _            .




                              CHAPTER4

                   CONCLUSIONS,RECOMMENDATIONS,

                        AND AGENCYCOMMENTS

CONCLUSIONS

       The Office of Education has recognized the importance
of study and evaluation     contracts  for providing  useful and
timely information,    making decisions    about the effectiveness
of its programs, and determining      the need for establishing
new programs.     If the Office of Education is to receive the
benefits   expected from the contracts,     actions need to be
taken to improve overall     contract administration.

       As stated on page 22, the Office of Education issued
guidelines     in June 1970 for use by contract        and grant man-
agement personnel.         These guidelines    were not, however,
sufficiently      comprehensive for use by project monitors in
determining      the procedures to be followed to monitor con-
tractors'    progress effectively.         At the close of our field-
work, HEWwas preparing         a guide for use by project monitors.
This guide can result        in a significant    improvement since it,
as tentatively      written,   deals with some of the problem ar-
eas identified      in our review.      For example, it points out
the need for clear work statements,           the need for contract
changes to be in writing,        and the need for effective      monitor-
ing.
FUXOMMENDATIONS
              TO THE SECRETARYOF HEW

       We recommend that,    in finalizing       the guide,   the Secre-
tary   of HEM provide for    inclusion     of:

       1. Guidance to help ensure that the objectives        and re-
          quirements of contracts     are clearly  understood by
          the contractors    and that the scope of work is de-
          scribed in sufficient    detail   in the contract.

       2.   A requirement to amend contracts,        in writing,   when
            changes are made.



                                   28
            .            .
                .I




                     .
    .
.       .
                             '   3. The methods to be used in monitoring         the contractor's
                                    progress during the period of performance.            Included
                                    in such methods should be the requirement          for site
                                    visits  at specified     points in the contract     period.
                                    As part of the monitoring       process on relatively
                                    large contracts,     the Office of Education should
                                    consider requiring      the use of a panel of experts to
                                    review periodically       the contractor's   progress and
                                    to recommend any necessary changes in scope, empha-
                                    sis, or level of effort.        Such reviews might be
                                    made on or before expenditure         of 25 percent of the
                                    effort  planned for the project        and at the halfway
                                    point,  as well as at the end before accepting the
                                    final product.


                                 4. Criteria    for use by agency personnel in evaluating
                                    a final    report prior to acceptance to determine
                                    whether    the contract objectives  have been satisfied.
                                 5. The steps required to be taken when considering     ac-
                                    tion against a contractor     for poorly performed work,
                                    including  consultation   with legal counsel and con-
                                    tracting  officials.

                      In addition,   we recommend that the Secretary provide
                for the establishment    of an orientation   course to acquaint
                agency program personnel involved in the administration          of
                study and evaluation    contracts  with the requirements      of
                Federal Procurement Regulations      and agency instructions.

            ACENCYCOMMENTS
                        The Assistant    Secretary,   Comptroller,   of HEW commented
                 on a draft of this report by letter         dated June 1, 1971.
                 (See app. I.)      He stated that the report indicated      that a
                 comprehensive review had been performed and that, although
                 limited as to the qpe of contracts          reviewed, the report
                 had identified     areas of improvement needed in the overall
                 administration     of contracts    and grants.

                       The Assistant Secretary informed us that HEW had in-
                 corporated  all the guidelines  and requirements recommended
                 by GAO in the guide for project managers which HEWplanned


                                                               29
                                                                       .        .
                                                                           I.




to publish during July     1971. In commenting on the need for
an orientation  training     course, the Assistant Secretary
stated that the Office     of Education had entered into a con-
tract for this purpose.       He said that a survey of training
needs had been completed      and that a course would be devel-
oped to start early in     fiscal year 1972.
       The Assistant Secretary stated also that the following
additional   actions had been taken or were planned by the
Office of Education to improve contract management.

     1. The goal of improving contract management was estab-
        lished as a primary component of the Commissioner's
        Management Objective for fiscal year 1972. Steps
        taken thus far to achieve this goal have been to
        (1) limit the authority         to obligate   the Government
        contractually      only to those persons formally desig-
        nated as contracting       officers    through the issuance
        of a personal warrant and (2) hire an experienced
        contracting     officer   to head the Office of Education's
        Division     of Contracts and Grants and provide him
        with top-management       support to accomplish his mis-
          sion.

     2,   Advance planning will be undertaken for the purpose
          of formulating  adequate scope of work specifications
          in future contracts.

     3. A modified      work statement is being incorporated      in
          the contract    for the study of the national    Follow
          Through program (see p. 16) to provide for orderly
          completion    of the contract.   Monitoring   of the con-
          tract is being elevated from branch to bureau level.
          At June 1, 1971, the Office of Education was nego-
          tiating   with the contractor   and expected to complete
          the modifications     by June 30, 1971.




                                  30
    .
.   .
                                      CHAPTER5

                                  SCOPEOF REVIEW

               Our review was directed   toward ascertaining    whether the
        policies   and procedures followed by the Office of Education
        in administering    study and evaluation    contracts  were ade-
        quate for ensuring that the information       obtained would serve
        the intended purpose effectively.        Most of our work was per-
        formed at the Office of Education headquarters        in Washing-
        ton, B.C.

              We initiaily    compiled a listing   of 86 study and evalua-
        tion contracts,    totaling   about $22 million,    from data fur-
        nished to us by the Office of Education and from listings
        prepared by GAO under a previous survey.         All of these con-
        tracts were to be completed after January 1, 1969.

              From the listing    we selected for examination      24 con-
        tracts totaling    about $11.3 million.     In selecting    these
        contracts,   we considered contracts    with a high-dollar      value,
        contracts  that had been completed, contracts       that had been
        active for relatively     long periods of time, and contracts
        awarded by a cross section of the various bureaus within the
        Office of Education.      We were concerned primarily     with those
        contracts  which required the contractor      to submit an end
        product in the form of a report,

                We reviewed the records for these 24 contracts          and dis-
        cussed the contracts      with agency officials       to determine
        whether the product produced or being produced by the con-
        tractor    was acceptable    to the Office of Education.        We then
        concentrated     our audit efforts     on seven contracts     where
        there were indications       either that the Office of Education
        was not satisfied     with the results      or that a&ion was neces-
        sary to improve the chances of receiving           the results   ex-
        pected.     We also interviewed     selected contractors      whose con-
        tracts were included in our review.




                                           31
        .. . .
.   .      *.
:
    .            .
.   .




                     APPENDIXES
                                                                                                APPENDIX I


             DEPARTMENT       OF   HEALTH,             EDUCATION.       AND     WELFARE
                               OFFICE     OF     THE     SECRETARY

                                   WASHINGTON,           D.C.   20201




                               .JuN 1 1971



Mr. Philip     Charam
Associate     Director
United States       General
   Accounting     Office
Washington,      D, C.

Dear Mr.    Charam:

The Secretary      has asked that I reply          to your letter     dated
February    19, 1971, with which you forwarded              the draft    report
of the General Accounting         Office    entitled     "Need for Improved
Procedures     for Administration        of Study and Evaluation
Contracts."       We appreciate    the opportunity        to review    and
comment on the report,         the conclusions       and recommendations.

The report    indicates    a comprehensive      review was performed        of
the administration       of the specific     types of contracts        examined
and weaknesses      in need of strengthening,         Although    limited     in
scope as to the types of contracts          examined,      the report
identifies    areas of improvements       needed in the. overall         admin-
istration    of contracts     and grants.

Detailed   comments on the recommendations    together with the
statements    of actions  taken or to be taken are set forth   in
the enclosures    hereto.

                                                         Sincerely            yours,




                                                         Assistant            Secretary,         Comptroller

Enclosures
Tab A - Contracting            Officer            Authority             in     the     Office     of
           Education
APPENDIX      I                                                                                      . . ..

                                                                                                               .:
 Department of Health,    Education   and Welfare Comments Pertinent   to                                     _ -
 the Draft Report to the Congress of the United States by the
 Comptroller   General of the United States on "Need for Improved
 Procedures  for Administration     of Study and Evaluation Contracts"


 OVERVIEW OF GAO REPORT

 GAO's report      indicates      that they believe        the Department needs to
 strengthen     its policies        and procedures      in administering          study and
 evaluation     contracts     let by the Office         of Education        (OE).     To
 accomplish     this,    they offer     recommendations        calling      for the
 Department to (i) incorporate             certain    criteria      relative      to contract
 administration       into its forthcoming          guide for project          managers, and
 (ii)   provide training        in contract      regulations      and administration
 for such program personnel.

 Our specific   comments on each of GAO's two recommendations                         - paraphased
 somewhat - follow:

 GAO RECOMMENDATION

        In its forthcoming         guide for project            managers, HEW should
       incorporate     (i) guidance on defining                 contract     scope and
       objectives,     (ii)     requirements          calling    for amending contracts
       in writing     whenever siqnificant               changes to the scope of the
       work are made, (iii)           information          as to the methods to be
       used in monitoring          contract        progress     on contract     objectives;
        (iv) criteria       for evaluating,           before final       acceptance,     the
       suitability     of final       reports       of contractors,         and (v) informa-
       tion on a contractor's            liability         for unsatisfactory        performance
       under the contract.

 DEPARTMENTCOMMENT

We have incorporated    all of the GAO recommendation   in the forthcoming
guide for project    managers.   We expect that the guide will   be published
during July 1971.

GAO RECOMMENDATION

       HEW provide      training   in contract         regulations      and administration
       for program      personnel.

DEPARTMENTCOMMENT

We concur     with   this   recommendation.

OE has entered    into a contract   with Harbridge    House, Inc. for a
management training    course.    A survey of training    needs has been com-
pleted and a curriculum     will  be developed for implementation     in early
fiscal year 1972.


                                                36
            .     ..                                                                      APPENDIX       I

.   .              1
        I

.       .

                GENERAL COMMENTSNOT RELATED TO SPECIFIC RECOFlMENDATIONS

                As recognized   by the GAO, the Office      of Education     has formal guide lines
                for project   monitors   which they will    continue   to follow,    except as noted
                below regarding    scope of work specifications,       until   the Department ‘S
                guide becomes effective.

                In addition,      OE has established        a goal for improvina        contract    management.
                The goal is a primary component of the Commissioner's                    Management Objec-
                tive for FY 1972.          It recognizes      the present   lack of effective        contract
                policies     and procedures        and makes provisions     to remedy the situation.
                Two steps have already            been taken toward achievement         of this goal.
                Effective     July 1, 1971, only those individuals             formally     designated     as
                Contracting      Officers     through the issuance of a personal warrant               as a
                Contracting      Officer     will   have authority    to obligate     the Government
                contractually       (Tab A).       Also, OE has employed from DOD a competent,
                experienced      contracting       officer  to head its Division        of Contractsand
                Grants who will        have top management support to accomplish               his mission.

                Regarding the General Accounting         Office    identification       of the difficulties
                encountered  with precise    specification       of the scope of work on contracts
                for studies  and evaluation    of complex educational             programs on pages 14
                through 19 of the report     under the heading "Need for !Iore Specific
                Description  of Work to be Performed",          OE will     engage in advance planning
                with the view to formulating      adequate scope of work specifications.

                Also, regarding      contract    OEC-0-8-522480-4633,       a longitudinal      evaluation
                of the national       Follow Through Program, discussed          in the same section       of
                the report,     OE is drafting     a modified     work statement     which will     be
                incorporated     into the contract.        This modification      will   be accomplished
                by June 1, 1971, and will         provide for orderly       completion     of the contract.
                Contract    monitoring     will  be elevated    from branch to bureau level,           and
                will   be accomplished      by the use of milestones.         A report     on this action
                will   be available     at OE for your review.




                                                         37
        APPENDIX I                                                                                                          . ...


                           DEPARTMENT         OF    HEALTH,         EDUCATION.       AND    WELFARE                                 .   .
                                                    OFFICE     OF   EDUCATION




  Date: March 15, 1971
R&y b
kt~ of: ADM,CGD

subject: Contracting       Officer      Authority       in the Office            of Education

   5:     Deputy Commissioners
          Associate/Assistant           Commissioners

          1,    Effective      1 July 1971, Contracting      Officer     authority     in the Office      of
          Education       shall be restricted       to those individuals      formally    designated
          as Contracting        Officers     through the issuance of personal          Contracting
          Officer's      warrants.       This policy   shall apply to all discretionary            grants
          and all contracts          and agreements.

          2. Bureaus requiring      the designation      of individuals    as Contracting
          Officers  shall make an individual       application     for each individual     to the
          Director  of Contracts    and Grants Division        by 15 April  1971.   Applications
          shall contain   the following   data:

                a.     Name, grade        and position          of the applicant.

                b.     A copy of his        position          description.

                c.     A reasonably        detailed   description  of those specific                          duties      that
                       require  his       designation   as a Contracting   Officer.

                d.     A reasonably detailed   description       of experience                        which     qualifies
                       him to act as a Contracting      Officer.

                e.     A statement        as to the desired               monetary      limitation      of the warrant.

          3. For the purnose of this policy,           contractual    action,     i.e.,   action that
          can be taken only by a Contracting          Officer    is defined as any action taken
          in connection     with a discretionary      grant,   agreement,     contract    or other
          instrument    which may obligate       the Government contractually.          It includes,
          but is not limited     to:

                 a.    Solicitation        of proposals.

                 b.    Synopsizing.

                 C.    Issuance       of letters       with     contractual          implications.

                 d.    Oral discussion  of contractual    matters                      with contractors,               grantees,
                       other than those matters     of a technical                      nature which fall               within
                       the scope of the grant/contract.

                                                              38
      _         .




          1 .




                                                                                                           APPENDIX I
..                  .


                        e.   Contractual          direction        to contractors/grantees.

                        f.   Change of work           scope.

                        8*   Time extensions.

                        h.   Contractual approval       of reports,              documents,     and materials        called
                             for by the contract/grant.

                        i.   Approval of labor and indirect                    cost   rates   (Exclusive      of   functions
                             under JEW jurisdiction).

                             Resolving       of    questions        of allowability       of costs.

     4. Any question                     as to the contractual    implication   of a proposed action
     by any individual                    not designated   as a Contracting    Officer  shall be referred
     to the Contracts                    and Grants Division   for resolution.

     5. Any action taken                          in violation          of the policy     set forth above will
     subject   the individual                        concerned        to disciplinary      action and possible
     pecuniary    liability.

     6. All existing                       delegations          of authority      are being    changed      to reflect
     this policy.




     Leon If, Schwartz
     Assistant  Commissioner                         for      Administration




                                                                       39
                                             OFFICE OF EDUCATION
                                                                                                                                . ...

                           SCHEDULE OF CONTRACTSSELECTED BY CA0 FOR REVIEW (note                                a)                              :
                                                                                                                                            .    .
                                                                                                                        Period
                                                                                                                           of
       Contract   number                                Purpose        of contract                                   performance        .

COMPLETEDCONTRACIS (note   d):
   OFGO-9-099007-4581                                                                                                 6-20-69      to
                                     Study    of special       adult    basic      education         projects        12-15-69


    OEC-O-8-001681-1882              Testing and further  development                  of an apera-
                                     tional  model for the evaluation                  of alternative                 2- l-68 to
                                     title  I projects                                                                7-31-6Se

    OEC-0-9-009001-1381              Perform analysis          of 1968 Survey of Compensatory                        12- 2-68 to
                                     Education                                                                        6- l-69

    OEC-O-V-089013-2471              A study of further      selected Programs for the
                                     education    of disadvantaged    children,    a study of
                                     selected    exemplary programs for vocational         edu-
                                     cation in secondary schools,        and field   testing
                                     a prototype    guide for authors of evaluation          re-                      l-27-69      to
                                     ports of educational      programs                                               6-26-69

   OEC-O-8-001714-1714               National     evaluation       of Project        Follow       Through             7-10-67      to
                                                                                                                      6-30-69



   OEC-O-8-080346-2800               An investigation,       analysis,  and evaluation    of
                                     activities     connected with the operation       of Edu-                        2-12-68      to
                                     cational   Information     Service Centers                                       6-11-70


   OEC-O-8-080468-3534               Study of selected relationships        between pupil
                                     and staff   and educational   facility     characteris-
                                     tics associated   with Public Law 89-10 title           I                        4- l-68      to
                                     projects  in Iowa                                                                4-30-69



  OEC-O-089017-3512(016)             Special dissemination project for programs fi-
                                     nanced under the Elementary and Secondary Edu-                                   3-26-68      to
                                     cation Act                                                                       3-25-69


 OEC-O-8-000082-4720                 Evaluation     of Teacher         Training      Institutes                       6-30-68      to
                                                                                                                      8-16-69


 OEC-l-7-071052-2808                 Study of National          Defense Education             Act,      ti-           .!I- 5-67 to
                                     tle IV Fellowship          programs                                              7-31-70


 OEC-O-9-089028-0710                 A study of the need for             educational          manpower                8-12-68      to
                                     for handicapped children             and youth                                   7-31-70

OEC-6-99-147                        To design,    test, operate,   and evaluate   Insti-
                                    tute Information    System for National     Defense                              12- 3-65 to
                                    Education Act institutes     for 1966                                             4- l-70



OEC-l-7-071211-4577                 TO study students graduated  from the public                                      6-22-67 to
                                    schools of (name of city deleted)                                                12-31-6Sf


OEC-O-S-980797-4634                 Establishment       of a National             Planning     Congress               6-30-68      to
                                                                                                                     12-31-69


                                         Total    completed       contracts
                                                                                                          APPENDIX II

                                  Amount of
                                contract  with
          5Pe     of              amendments                  Usefulness of end product
        ‘contract                   (note b)                 and other comments (note c)

                                                 The final      product was disseminated    to State
Fixed price                      s     24,750    agency    officials    and through the Educational
                                                 Resources Information        Center (RRIC) system.
                                                 Model developed was considered   satisfactory,
                                                 but full-scale  usage is not envisioned.       The
Fixed     price                        58,990    report was disseminated  through the ERIC sys-
                                                 tem.
                                                 The final report was considered of limited       use-
Fixed     price                        80,579    fulness;   however, certain  analyses were useful
                                                 for planning the new direction     of the analysis.




                                                 The final report was placed into the ERIC SYS-
Cost plus fixed           fee         134,688    tern, and the contractor published the report.
                                                 The report was considered    of limited  usefulness.
Cost reimbursement                    146,100    It iras used, however, as background material
                                                 for a subsequent evaluation.      It was never dis-
                                                 seminated to the public or placed in the ERIC
                                                 system.
                                                 The Directory  of Information    Centers was con-
                                                 sidered very acceptable.      It was used inter-
Cost plus         fixed   fee         277,608    nally as a reference document and was placed
                                                 into the RRIC system; also, it is being pub-
                                                 lished by a commercial firm.
                                                 The final    report was never submitted because
                                                 the contract     was terminated,    but data received
                                                 from the contractor       were considered useful.
Cost     reimbursement                 79,034    The data were used in-house to supplement in-
                                                 formation    on title   I. The contractor    received
                                                 $30,000 of the contract       amount prior to its
                                                 termination.
                                                 Articles  were considered very usable material.
                                                 They were disseminated   through educational   mag-
Cost reimbursement                    100.000    axines and the ERIC system.     In addition, they
                                                 have been incorporated   into special Office of
                                                 Education publications.
                                                 Product was considered very useful for deci-
Fixed price                            69.469    sionmaking purposes.     It helped establish
                                                 guidelines  for the following    year’s institutes.
                                                 It was disseminated   through the ERIC system.
                                                 Report was considered extremely        useful for de-
cost     plus fixed       fee         135,948    cisionmaking     purposes.   It till   be wed to es-
                                                 tablish   guidelines    for programs   and will be in-
                                                 serted into the ERIC system.
                                                 Product was being wed as background informa-
Cost reimbursement                    430,487    tion for future programs.    The general plan was
                                                 disseminated   through the RRIC system.
                                                 Report data were considered useful,          but interim
                                                 data were too late to be used for planning pur-
Cost reimbursement                    404,880    poses.    Information     has been extracted    from
                                                 data for inclusion      in Office of Education pub-
                                                 lications    and reports;    also, date mere dissem-
                                                 inated to the public upon requests.
                                                 The end product was considered to be of no use
Cost reimbursement                    103,172    to the Office of Education.   In addition, it
                                                 was never placed into the RRIC system or dis-
                                                 seminated publicly.
                                                 This report was criticized     as containing    little
Cost reimbursement                    200,000    or nothing original;    however, it was used for
                                                 program planning in-house.      Further,    it was
                                                 disseminated   through the ERIC system.
                                 S 2.245.705


                                                                 41
                                                                                                                                                 *        .

                                                                                                                                                     I.




                                                                                                                                                                        .




                                                                                                                                             .                         ,.
                                                                                                                                       Period                      '
                                                                                                                                          of
        Contract    number                                             Purpose     of contract                                      performance*

ONGOING CONTRACTSbate         9) :
   OEC-D-B-080431-2999                        Study     the use and effectiveness      of title                III                   3- l-68                  to
                                              funds     in selected developing    institutions                                       2-28-69


    036-5-85-038                              An integrated,            longitudinal         study   of practical                    6- l-65                  to
                                              nursing                                                                                6-30-69


    OEC-5-10-045                              An evaluation     of the effectiveness      of an early
                                              enriched    school curriculum     in reversing   the ef-
                                              fects of environmental       conditions   which lead to                                7- 1-64                  to
                                              failure  in school                                                                     6-30-70


    OEC-5-10-219                              A cross national    study             of socialization          into                   3- l-65                  to
                                              compliance   systems                                                                   3-31-69

    OEC-l-6-061774-1887                       Study     of school        integration                                                 6-27-66                  to
                                                                                                                                     9-30-69

    OEC-09-099004-3332                        Development of a research design for a compara-
                                              tive longitudinal  study of Demonstration Educa-                                       3-12-69.to.
                                              tion Progrsms                                                                          7-31-70
    OEC-l-7-070992-5022                        Survey and analysis               of educational         information                  6-30-67                  to
                                                                                                                                    10-30-70

    OK-0-0-522460-4633                         Longitudinal  evaluation                of the national        Follow                 b-27-68                  to
                                               Through program                                                                      12-31-70
    OEC-O-0-000310-3742                        A study of resources and major holdings    avail-
                                               able in U.S. Federal libraries maintaining     ex-                                    6-20-68                  to
                                               tensive or unique collections                                                         g-19-70
    OEC-O-9-402025-3726                        Evaluation  of a program to train teachers to
                                               manage social and emotional   problems in the                                         6- 2-69 to
                                               classroom                                                                             2-28-71
                                                      Total    ongoing      contracts
                                                      Grand total
aContracts    selected    had a completion      date    subsequent         to January        1, 1969.
bContract    amount computed     through     June 30, 1970.
CComments in this column are based on Office                  of Education         records     and on discussions            with   Office           of
 Education officials.

dContracts   whose final     reports   were submitted          prior     to September         1, 1970.

%he Office of Education    informally          extended       the contract         to May 1969 at which               time   the contractor
 submitted the final  product.
fl'he Office of Education  informally  extended the period of performance  to March 1969.                                       Nevertheless,
   the contractor did not submit the final   report until September 11, 1969.
gAugust 31, 1970, was used as a cutoff  date for our fieldwork.   However, we contacted   Office of Edu-
 cation officials in June 1971 to obtain the current   status of these contracts  and were informed  that
 eight had been completed and two--OEC-S-85-038  and OF&O-8-522480-4633--were    still  ongoing.
                                                                                                              APPENDIX II
I *
*‘ .            .                       Amount of
 - .                                  contract  with
                Typeof                  amendments                  Usefulness of end product
                                                                   and other comments (note c)
       .-      contract                   (note b)


       Fixed price                     8      74,461   Report   was 18 months overdue     at August     31,
                                                       1970.

       Cost reimbursement                    395,393   Report   was 14 months overdue     at August 31,
                                                       1970.



       Cost reimbursement                    542,180   Report   was 2 months overdue     at August     31,
                                                       1970.
                                                       Report   was 17 months overdue     at August     31.
       Cost reimbursement                    300,000   1970.
                                                       Report   was 11 months overdue     at August     31,
       Cost reimbursement                    209,026   1970.

                                                       Report   was 1 month overdue     at August     31,
       Fixed    price                        254.023   1970.

       Cost reimbursement                     55,890   Report   due after   August   31, 1970.

       Cost plus        fixed   fee        7,028,805   Report   due after   August   31, 1970.


       Cost reimbursement                    137,118   Report   due after   August   31, 1970.


       Cost plus        fixed   fee          111,618   Report   due after   August   31, 1970.
                                           9.108.514
                                       $11,354.219




                                                                       43
APPENDIX III                                                          , :               I



                                                                                    :
                                                                  .
                                                                            3   ’


                PRINCIPAL OFFICIALS OF THE                                  .           I




        DEPARTMENTOF HEALTH, EDUCATION, AND WELFARE
        RESPONSIBLEFOR ADMINISTRATION OF ACTIVITIES
                 DISCUSSEDIN THIS REPORT

                                        Tenure of office
                                        From            -To
SECRETARYOF HEALTH, EDUCATION,
  ANDWELFARE:
    Elliott L. Richardson            June   1970   Present
    Robert H. Finch                  Jan.   1969   June 1970
    Wilbur J. Cohen                  Mar.   1968   Jan. 1969
    John W. Gardner                  Aug*   1965   Mar. 1968
    Anthony J. Celebrezze            July   1962   Au%* 1965
ASSISTANT SECRETARY(EDUCATION):
    Vacant                           June   1970   Present
    James E. Allen, Jr.              bY     1969   June 1970
    Peter P. Muirhead (acting)       Jan.   1969   WY      1969
    Lynn M. Bartlett                 July   1968   Jan. 1969
    Paul A. Miller                   July   1966   July    1968
    Francis Keppel                   Oct.   1965   &Y      1966
COMMISSIONEROF EDUCATION:
   Sidney P. Marland, Jr.            Dec.   1970   Present
   Terre1 H. Bell (acting>           June   1970   Dec. 1970
   James E. Allen, Jr.               %Y     1969   June 1970
   Peter P. Muirhead (acting)        Jan.   1969   &Y      1969
   Harold Howe II                    Jan.   1966   Dec. 1968
   Francis Rappel                    Dec.   1962   Jan. 1966




                                                   U.S GAO. Vasb., D.C.


                                44
              ‘.,

.         ’

    . t
    -




                     Copies  of this report are available       from the
                     U. S. General   Accounting   Office,    Room 6417.
                     441 G Street, N W., Washington,      D.C., 20548.

                      Copies     are provided     without        charge       to Mem-
                      bers of Congress,         congress        iona I committee
                    I staff members,     Government         officials,       members
                      of the press,     college     libraries,        faculty    mem-
                      bers and students.         The price to the general
                      public    is $1.00 a copy.        Orders should be ac-
                    1 companied     by cash or check.