oversight

Administration of Cost-Plus-Award-Fee-Type Contract for Aircraft Maintenance Support Services

Published by the Government Accountability Office on 1971-04-14.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                                               llllllllllllllllllllllllllllllllllll
                                                      LM095585




A~dministration Of Cost-Plus-
    rd-Fee-Type Contract
For Aircraft Maintenance
Support Services                 B-   133394




Manned Spacecraft       Center
National Aeronautics     and
  Space Ad ministration
CIVIL      DlViSlON




                  B-133394

                  Dear    Dr.   Low:

                          The accompanying        report    presents     the results    of our review         of
                  the administration       of the cost-plus-award-fee-type            contracts     with
                  Dynalectron      Corporation     for aircraft    maintenance       support    services       at
                  the Manned Spacecraft         Center,   Houston,      Texas.

                            In response      to our previous      report    on our review    of the admin-
                  istration     of a cost-plus-award-fee-type             contract    for computer      pro-
                  gramming        and operational        support   services     (B-133394,   December        18,
                  19691, the National         Aeronautics      and Space Administration         furnished      to
                  us its overall      concepts      and views on award fee contracting.              Matters
                  in this report      have been discussed          with officials    of the National     Aero-
                  nautics     and Space Administration           and of the Manned Spacecraft            Center.
                  Any additional       written     comments      on the specific     matters   discussed       in
                  this report      and on the actions to be taken by your agency will be ap-
                  preciated.

                          We are recommending              that the National   Aeronautics      and Space
                  Administration        examine      into the work requirements        of the aircraft
                  maintenance       contract     with a view to establishing       a performance       eval-
                  uation plan which will place greater               emphasis   on objective    performance
                  standards      in terms     of output criteria      and that it make use of available
                  performance        indicators.       In the event that the National      Aeronautics      and
                  Space Administration           concludes     that such standards     cannot be devel-
                  oped, we believe that contracting              for aircraft  maintenance      on a cost-
                  plus-award-fee         basis should be reconsidered.

                          Your attention    is invited   to section    236 of the Legislative      Reor-
                 ganization     Act of 1970 which requires        that you submit written        statements
                 of the action taken with respect          to the recommendation.        The statements
             ?r;
        :i ’  ‘i are   to  be  sent  to the House    and  Senate    Committees     on Government       Op- -2‘
                 erations     not later than 60 days after the date of this report            and to the
                 House and Senate Committees             on Appropriations      in connection     with the L,‘L’ .’




                                              50TH ANNIVERSARY           1921- '1971
B-133354




first  request      for appropriations             sutrrittt-d         I:)- yxr      agency-     rr1Dre   thar,
 60 days after       the date of this          report.


       Copies       of this    report    are being  sent to the Director,     Office                        of
Management          and Budget,       to the House   and Senate   Com-nittees       on
Government         Operations,        and to the House   and Sccate    Committees                           on
Appropriations.

                                                                 Sincerely        yours,




                                                                 Director,        Civil    Division


The Honorable       George          M. Low
Acting Administrator,               National      Aeronautics
   and     Space   Administration




                                                          2
                                      Contents
                                                                     PaFe

DIGEST                                                                 1

CHAPTER

   1      INTRODUCTION                                                 3

   2      NEED TO REVISE PERFORMANCE EVALUATION PLAN                   Q
              Lack of objectivity        in MSC's evaluation
                 of Dynalectron's     performance                      6
              Feasibility     of establishing     an objective
                 evaluation    plan                                   10

   3      FEE DETERMINATION ACTIONS THAT .INCREASED
          CCNTRACTGR'S FEES                                           14
              Deviations    from established         award
                fee distribution        guide                         14
              Changes in recommended scores not
                supported      by factual     information             15

   4      CCNCLUSIONS AND RECOPalENDATIONS                            17
              Conclusions                                             17
              Recommendations to the NASA Adminis-
                trator                                                17

   5      SCOPE OF REVIEW                                             19

                                 ABBREVlATIONS

CPAF      Cost   plus    award    fee

MSC       Manned Spacecraft           Center

NASA      National      Aeronautics       and Space Administration
                                                   ADMINISTRATION
                                                               OF COST-PLUS-AWARC-
REPORT TO THE ADMINIST-%T3fi                       FEE-TYPE CONTRACT FOR AIRCRAFT
NATIGNAL AERONAUTICS ANC                           MAINTENANCE SUPPORT SERVICES
SPACE AiXINIST.Z4?IO,'v                            Manned Spacecraft      Center
                                                   National   Aeronautics    and Space
                                                     Administration     B-133394


DIGEST
------

WHY THE REVIEW WAS MADE

    During fiscal    year 1970 the Manned Spacecraft      Center administered        23
    support  services    contracts  totaling   about $110 million.        Of these con-
    tracts,  13 totaling     about $81 million   were cost-plus-award-fee        con-
    tracts.

    Cost-plus-award-fee          contracts    are used extensively         at other National
    Aeronautics       and Space Administration           (NASA) locations.       This type of
    contract    provides     for payment to the contractor            of a variable       fee de-
    termined    subjectively       by Government evaluators         on the basis of pe-
    riodic    evaluations      of the quality       of the contractor's       performance
    against    criteria     set forth      in the contract.       The contract      provides   for
    a base fee, which may be zero, and for an upward adjustment                        of the fee
    which is called an award fee.               After    the award fee has been determined,
    it is not subject        to the disputes         clause of the contract.

    The General Accounting       Office    (GAO) reviewed Manned Spacecraft            Center
    contracts   with Dynalectron       Corporation   for aircraft       maintenance     support
    services   to determine    the adequacy of award fee evaluations              and the
    need for and feasibility        of establishing     objective      standards    for con-
    tractor   evaluations    on the basis of contractor          performance     or output.


FINDINGS AND CONCLUSIONS

    For the most part,      objective   measurement standards,      as set forth    by
    NASA's cost-plus-award-fee        contracting   guide, had not been developed
    for use in evaluating       the contractor's    performance.      Consequently,
    award fees were paid to the contractor          primarily    on the basis of
    opinions   and judgments      of Manned Spacecraft     Center evaluators     as to
    the contractor's    performance.        (See p. 6.)

    GAO's review       showed that guidance on aircraft       maintenance   operations
    is available       from Air Force sources and that Manned Spacecraft           Center
    personnel    are     sufficiently     involved in the details   of maintenance
    management to        establish    work output measurement standards     for objective




                                          1
evaluation       of wlletkr*  hnth the quality    and the timeliness      of the con-
tractcr    '5 'tioi*k dre iq accordance with     HASA's cost-plus-award-fee      con-
tracting     guide.      (See pp. 10 and 11.)

GAO found that deviations         from established     award fee distribution
guides and unsupported         award fee adjustments     by higher level evaluators
resuited  in higher fees being paid to the contractor.              GAO concluded
that in certain    instances      the contractor   received    an award fee for ex-
cellent  performance     when3 in fact,      its performance    was supportable   at
only the high-good      level.      (See pp. 14 through 16.)




GAO recommends that NASA examine into the work requirements                  of the air-
craft   maintenance     contract    with a view to establishing        a performance
evaluation    plan which will       place greater    emphasis on objective       perfor-
mance standards      in terms of output criteria          and that it make use of
available    performance     indicators.      In the event that NASA concludes
that such standards       cannot be developed,       we believe    that contracting
for aircraft     maintenance      on a cost-plus-award-fee       basis should be re-
considered.      (See p0 17.)




Manned Spacecraft      Center officials      with whom GAO discussed       this matter
expressed   a willingness      to explore    the possibilities      of work order
evaluations   based on output measurement standards.               They noted, however,
that such evaluations       represent    a significant      change from the present
evaluation   plan and that the present staffing             levels may not be adequate
to implement the change.         (See p. 38.)
                              CHAPTER1

                            INTPODUCTION

        The General Accounting Office made a review of the
cost-plus-award-fee       (CPAF) contracts    with the Dynalectron
Corporation     for aircraft   maintenance support services pro-
vided to the National Aeronautics          and Space Administration's
Manned Spacecraft Center (MSC), Houston, Texas,              The pur-
pose of our review was to determine the adequacy of award
fee evaluations      and the need for and feasibility        of estab-
lishing    objective   standards for contractor      evaluations    on
the basis of contractor       performance or output.

       A CPAF contract  is a cost-reimb,ursement     type of con-
tract.    According to NASA, it is appropriate       for 'use in sit-
uations where it is difficult       to specify all contractual
requirements    prior to negotiation    and is particularly     suit-
able for use in contracting      for support and nonpersonal
services,

        The CPAF type of contract   provides that the contractor
be paid a variable     fee determined subjectively         by desig-
nated Government personnel on the basis of periodic             evalua-
tions of the quality      of the contractor's     performance against
criteria    set forth in the contract,        This type of contract
provides for a base fee, which in some contracts             may be set
at zero, and for an upward adj,ustment of the fee, called an
award fee.     Once the award fee is determined,         it is not
subject to the disputes clause of the contract.

       For about 3 years prior to October 29, 1965, MSC ob-
tained its aircraft      maintenance support services from the
Air Force.     The maintenance services were actually      provided
by the Dynalectron      Corporation   under an Air Force time-and-
materials-type     contract.     From October 29, 1965, through
October 31, 1966, MSC obtained these services directly          from
Dynalectron    under a CPAF contract     which had been competi-
tively   negotiated.     From November 1, 1966, through Octo-
ber 30, 1969, these services were obtained from Dynalectron
under CPAF contracts       which had been negotiated   on a sole-
source basis.       From October 31, 1969, to October 30, 1970,


                                    3
Dynalectron       provided     aircraft    maintenance     services   under a
competitively        negotiated       CPAF contract,      Concerning   the
contract      for the period        October 31, 1970, through        October 31,
1971, these services           were negotiated       with Dynalectron     on a
sole-source       basis,

        The estimated      cost     and fees     paid   are    set   forth    below.

  Year                                                     Award fee
 ending        Estimated            Base        Maximum      Amount          Percent
October           cost              fee         amount        paid             paid

     1966     $1,X3,956           $12,500       $ 60,000      $ 53,216           89
     1967      2,353,ooo                    1    P40,OOO       1'94,000          81
     1968      2,747,680              -          166,520       121,935           73
     1969      2,935,972              -          189,400       176,360           93
     1970      3,275,ooo              -          203,000          (a>            -
     1971      3,568,130              -          237,000          (b)            -

aThe award fee available    for the first    and second 4-month
 evaluation   periods was $135,334,    of which the contractor
 received   $106,550,  or about 79 percent.
b
    At the time of our review   at MSC, the first evaluation   pe-
    riod was not completed,   and no award fee had been deter-
    mined,

       The specific     support    service     functions    provided     by
Dynalectron    included     service,     maintenance,      modification,
and ground support       of aircraft       assigned     to MSC'S Aircraft
Operations   Office.      The following        aircraft    were assigned
to MSC on June 30, 1969.

             Airplanes                                               38
             Helicopters                                              4
             Lunar landing          training
                vehicles                                             -2
                   Total

       The NASA Cost Plus Award Fee Contracting    Guide
(NHB 5104,4)   dated August 1, 1967, provides   contracting
guidance   and describes  NASA's policies  and concepts

                                            4
concerning the 'use and administration         of CPAF contracts.
The guide points out the necessity       for a performance evalu-
ation plan which includes       (1) the criteria    against which
the contractor's     performance will be evaluated;       for example,
the timeliness    with which particular      types of jobs are com-
pleted,    and (2) the standards of measurement to be applied
to the selected criteria;       for example, the number of times
particular    jobs are completed on time as compared with the
number of times they are delayed.

       Evaluation    of Dynalectron's     performance was accom-
plished by an Award Fee Evaluation           Committee and an Award
Fee Evaluation     Board.   The committee met monthly and its
members developed a consensus evaluation,             including   scores.
Periodically,     the committee prepared,        for presentation     to
the board, a consolidated        evaluation    report which included
scores and a s,uggested fee.         This report,     initially   pre-
pared every 3 months, is now prepared every 4 months.                 The
board considers the committee's          presentation      and makes its
recommendation to an official         who decides on the award fee
to be paid to the contractor.
                                       'X4PTER
                                       -.-          2

            NEED TO REVISZ PEXFORHANCE EVALUATION PLAN

       Cur rczviek.i showed that MSCQs performance              evaluation
plans for aircraft        maintenance     support    services,      for the
most part,     did not include      objective     standards      by which the
contractores      level   of performance      could be measured to de-
termine     the award fee.       The omission     of objective        measure-
ment standards       is contrary    to instructions         in NASA's CPAF
contracting     guide,    which requires      that objective        standards
be provided     for evaluating      the performance         of contractors.
We found that2       in the absence of objective            standards,      award
fees had been paid to the contractor              primarily      on the basis
of opinions     and judgments      by MSC's evaluators.

LACK OF OBJECTIVITY IN M!TC' S
EVALUATION OF DYKALECTRQNsS PERFORMANCE

        Although      NASA's CPAF contracting               guide describes          a
CPAF contract         as one with a variable              fee determined          by sub-
jective      evaluation     of the eontractorls               performance,        it pro-
vides that such subjective               evaluation         include     objective
measurements        whenever    such standards           are appropriate            and
can be defined.          The   guide     states     that,      for   the   satisfactory
operation      of the evaluation           procedures,         the performance
eval.uatFon      plan   should    clearly      define       input    and output--
input evaluation         being largely         subjective          and output       evalua-
tion being objective.

         NASA's CPAF contracting         guide adds emphasis by stating
that the evaluation         plan must clearly          define    performance
criteria     in terms which express            results    rather   than causes
or kinds     of effort.       In addition,        the guide states        that an
essential      element required      to ensure a realistic             evaluation
of contractor       performance    is the standard           of measurement       to
be applied       to the selected     criteria.

        The CPAF contracting     guide states      that the key elements
missing    in some early     CPAF contracts      were the standards    by
which contractor     performance      criteria    were to be evaluated.
The guide states     also that,     in preparing      CPA? contracts,    the
project    office should provide       objective     criteria,  whenever


                                             6
possible,      and a complete           and comprehensive    set            of   standards
against      which to measure           contractor  performance.

       In its      discussion   of evaluation               standards,        the   CPAF
contracting        guide states   that:

        "The observation       that some objective       criteria       and
        standards     cannot be generated      is difficult        to de-
        fend. *** For nonhardware-oriented            results,       if a
        sound description        of what constitutes        acceptable
        work cannot be described,         there should be no ef-
        fort    to incentivize     the performance."         (Under-
        scoring    supplied.)

        The MSC evaluation         plan for the 1st contract                year in-
cluded objective         measurement        standards     for two criteria--
control     of wages and salaries             and control      of employee turn-
over o The evaluation          plan also included             procedures       for
translating       the performance         into a numerical          score.       The Zd-
and 3d-year       plans included        measurement       standards        for only
the employee turnover          criteria.         The evaluation          plan for the
4th contract        year had no objective           measurement        standards
for any of the evaluation             criteria.       During the 5th contract
year,     measurement     standards       were developed         for about 85 per-
cent of the technical          work performance           criteria,       which con-
stituted     about 34 percent         of the total        performance         score,
although      they were not included            in the performance            evalua-
tion plan.

         The evaluation          plans for the 1st and 2d years included
a number of overall              performance           criteria,        such as (1) air-
craft     operationally          ready rates,            (2) aircraft        not opera-
tionally       ready--supply          rates;       i.e.,       the percentage       of time
aircraft       cannot be flown because of delays                         in obtaining     an
item of supply,           (3) aircraft           not operationally            ready--
maintenance        rates;      i.e.,     the percentage              of time aircraft
cannot be flown because of delays                          in completing        maintenance
requirements,          (4) aircraft          flight        hours,     and (5) flight
aborts.       However,       no objective            standards        for evaluating       the
contractor's         performance         on these criteria               were included       in
the evaluation           plan.       These criteria              were dropped from the
evaluation        plan during         the 3d contract              year and reestab-
lished,      in addition         to others        which were developed,               for the
5th year,        as discussed         above.

                                               7
        The eval.uation      plans for the 1st and 2d contract                  years
recognized    the existence          of some industry       standards       for
evaluating    performance         but indicated      that the standards
would not be used to measure the contractor's                      performance.
Both of these pl.ans included              the statement     that,     to the
maximum extent       practicable,        the contractor      would be evaluated
on an objective       basis,      using quantitative        analysis;       for ex-
ample, the number of tasks accomplished                  compared with the
number scheduled        or the number of work units             performed        com-
pared with the schedule            requirements.        Subsequent       plans did
not contain     such language.

        We noted that the following        statistics       on overall   con-
tractor      performance    for each evaluation       period were devel-
oped at MSC for use by the committee              and were evaluated
subjectively        without   benefit of objective      measurement    stan-
dards.

       1. Average number                of "squawks5  (deficiencies                    noted by
          MSC inspectors)                for each major or minor                    inspection.

       2. Operationally            ready       rates.

       3. Functional            check     flight          rejection      rates.

       4. Abort      rates.

       5. Repeat       squawks.

       6. Not operationally                ready--maintenance                  rates.

       7. Not     operationally            ready--supply              rates.

       8. Total      aircraft       flight             hours.

        Our analysis      of these statistics       and the performance
scores for each evaluation           period    showed that there was no
correlation      between them.       When there was an improvement           in
the statistics       on overall     contractor    performance,      there was
not a relatable       corresponding       improvement    in the perfor-
mance scores.        Furthermore,     the committee      chairman     told us
that other available         data on contractor       output    could not be
related     to the performance       scores.


                                                   8
         We discussed      the basis for Dynalectron's             performance
scores with the chairman              of the Award Fee Evaluation             Com-
mittee.      He told us that the scores were based on contrac-
tor performance         in the three operating         branches       of MSC's
Aircraft      Operations      Office.     He said that the performance
scores in these branches              were weighted    but that no percent-
ages or formulas          were used in establishing           the weight       for
each of the branches.              He said,however,      that about 85 per-
cent of all flying          was by the astronauts          and that 90 per-
cent of the score was based on the contractor's                       performance
in this area.         According       to the committee       chairman,      the key
factors     considered      were the flight-abort          rates,     the overall
aircraft     condition,       and Dynalectron's      response       to unusual
situations.




                                         9
FEASIBILITY OF ESTABLISHING AN
OBJECTIVE EVALWTIQN PLAI!: -

        NASA's CPAF contracting      guide states t'hat, as work
progresses under a CPAF support servicescontract,            an in-
creasing number of work direction         documents relating    to the
overall    task assignments can be subjected to objective           eval-
uation.      The guide states also that, if work direction         doc-
uments can be prepared, they should be used as the basic
work unit to be evaluated.         It states further    that, at the
end of each evaluation      period, the completed work orders
could be reviewed and the contractor's          performance could be
determined on the basis of how the contractor           completed each
work order.
      We noted that, although work direction   documents were
being issued by MSC to request work under the contract,     they
were not being utilized   by MSC evaluators  in their evalua-
tions of Dynalectron's   performance.

        When MSC determines that an aircraft      needs service,
Dynalectron     and MSC representatives     hold a preinspection
planning meeting and agree on a time that the aircraft             can
be made available      for maintenance and on the work to be
performed.      The work and the time it requires      is listed    on
the Pre-inspection      Planning Meeting form.     Any additional
work found necessary by the mechanics, supervisory            personnel,
or the pilot of the aircraft        is recorded on an Aircraft      In-
terflight     Worksheet or a Maintenance Discrepancy Work Rec-
ord.      These two documents and the Pre-inspection      Planning
Meeting form are the basis for all work performed on an air-
craft.

        In meetings with officials     at NASA Headquarters and at
MSC concerning our draft report,        we discussed the feasibil-
ity of using these work direction         documents in establishing
an objective      evaluation   plan.  We were told that standards
for output evaluation        were not available  and could not be
readily    established.

      We subsequently  consulted Air Force maintenance offi-
cials dealing with T-33 and T-38 training    aircraft. We
also performed additional    review work at MSC. Air Force
maintenance officials   informed us that a significant degree

                                  IO
of guidance on aircraft     maintenance operations   is available
from Air Force sources.      We noted that MSC management and
inspection  personnel and t'he contractor    were involved in
the details  of maintenance management. We believe that
standards for work output measurement could be established
to enable objective    evaluation   of both quality  and timeli-
ness of the contractor"s     work,

Evaluation   of quality      of contractor's
                                     ___I~-       work

       NASA's aircraft maintenance operation             at MSC is an ad-
aptation  of the basic Air Force installation              aircraft main-
tenance operation.

       MSC employs about 31 maintenance inspectors      who, inde-
pendently of any control by the contractor,        examine every
prescribed   maintenance operation   and establish    that it has
been completed.     We were told that these maintenance inspec-
tors were generally    men who had a thorough knowledge of the
various maintenance operations,     gained through firsthand
experience as aircraft    mechanics and maintenance supervi-
sors *

        Each maintenance operation        performed by the contractor
is reviewed by these NASA inspectors.              If the work is sat-
isfactory,    the inspector     indicates     his approval on the work
direction    document.    If the work is unsatisfactory,         however,
no indication     is recorded on the work direction         document.
No other record is maintained          of the results    of these in-
spections.      In our opinion,     a complete record of each MSC
inspection     should be maintained       to provide a source of in-
formation    that could be used to evaluate the quality           of the
contractorPs     work.

Evaluation   of timeliness      of contractor's      work
      Air Force time standards for T-33 and T-38 aircraft
maintenance operations     are available,    and they could be
adapted to the needs of l%Xvs local operation.          We noted
that three distinct    categories    of maintenance are performed
on a repeated basis on T-33 and T-38 training        aircraft   at
MSC; these are (1) maintenance performed in accordance with
Air Force Periodic   Inspection    Work Cards, (2) maintenance
performed in accordance with Air Force Time Compliance Tech-
nical Orders, and (3) corrective       maintenance.
       Ue w~7iewed   MSC"s aircraft     ut~%izatisn     records for the
23 T-38 trzinin&     aircraft    on which maintenance work had been
performed by the contractor        during the 15-month period from
June 1968 through August 1969. We noted that about 33 per-
cent of tne contractor's       time w3.s spent cm maintenance pre-
scribed by the Periodic Inspection            Work Cards, about 14 per-
cent was spent on maintenance prescribed             by the Time Compli-
ance Technical Orders, and the remaining 53 percent was
spent on corrective      maintenance,      including   some aircraft
modification   work.

        Maintenance prescribed   by the Periodic     Inspection   Work
Cards is performed on each of the aircraft        after every 400
flying hours.      It consists of specific    inspection     and main-
tenance operations     performed on the basis of a task-by-task
listing    on each of the work cards.      The work cards pre-
scribe a standard work time for tasks listed          on the card as
well as a total work time for each of the work cards.
These time standards are prescribed        by the Air Force as a
guide in determining     the work time normally needed to com-
plete each of the 400-hour inspection        and maintenance oper-
ations e

       Naintenance based on the Time Compliance Technical Or-
ders must be completed on each of the aircraft         within a
specified    period or the aircraft   must be restricted      or
withdrawn from service for safety reasons, until          compliance
with the technical      order is accomplished.   We noted that
the technical     orders prescribed  (1) the detailed     work
phases to be accomplished and (2) the standard man-hours
for completing the requirements      specified by each of the
technical    orders.

        The third category consists     of corrective    maintenance
which is performed when a specific         problem arises on an
aircraft.      After agreement between the contractor       and MC,
the particular      maintenance operation     decided on is noted on
the work direction       document by reference to the number of
the particular      Air Force technical   manual detailing     the
prescribed     procedures for the completion of the work.
     We noted that, at one Air Force installation     at which
T-38 aircraft  were being maintained, local job standards
based on local conditions  had been established   for many of

                                  112
the corrective  maintenance operations.   We were told that
various guidance data could be obtained from both field and
Washington headquarters   of the Air Force to assist in the
establishment  of local standards for maintenance operations.

       We believe that, with MSC's experience in aircraft
maintenance and with available     Air Force time standards,
MSC could establish    objective  standards for measuring the
timeliness    of the contractor's  maintenance work.




                               13
                                     C-HAPTER 3
                                     ---

                     FEE DETERMINATION ACTIOn'S THAT

                        INCREASED COWRACTOR'S -.-FEES

        During our review we noted several             instances      of devia-
tions    from established      fee distribution        guides and of unsup-
ported     adjustments    which increased       Dynalectron's       fee.   We
believe      that the deviations     arid unsupported        adjustments     re-
sulted     from the lack of objective          standards     for use in eval-
uating     performaacc    and that they illustrate           the need to es-
tablish      standards  for measuring      contractor      performance.

DEVIATIONS FROM ESTABLI§H.ED
AWARD FEE DISTRIaUTION GTJIDE

         During the third       quarter     of the 1st contract        year,    the
Award Fee Evaluation           Committee     recommended a deviation          from
the established         award fee distribution           guide which resulted
in the payment of a higher              fee to the contractor.          The com-
mittee      recommended that the fee for the third                and fourth
quarters       be computed in accordance          with the contract        fee
distribution        guide that had been negotiated             for the 2d con-
tract     year,     We were unable        to determine      the reason for the
use of the 2d year! s fee distribution                 guide to compute the
fee for the 1st year,            The committee's         use of this guide re-
sulted      in tl-ce computation     of a higher       fee than would have
been computed under the award fee distribution                     guide nego-
tiated      for the 1st contract         yearc,

       After   the committee?     presentation    to the Award Fee
Evaluation     Board concerning    past evaluations      and award fee
earnings,    the board accepted      the committee's     recommendation.
The result     was that Dynalectron      received    an additional    award
fee of about $2,860,

       We noted that,        during    the board meeting           to consider     rec-
ommendations      concerning       the third      quarter     of the 1st contract
year,    one board member requested             the committee        to recommend
a special     end-of-contract-year            bonus.      According     to the min-
utes of the board meeting,             the purpose of the bonus was to
increase    the contractor"s          overall     fee earnings       even though
it was recognized         that the contractorss            overall     performance
had not been excellent,
       The    committee                      a ?>o!iuS AVarCl Lee or Sl&,ZZG
                             :eCo!!:‘i!C?;li;?O


for the 1st contract          year,       Records -made a;7ailable     to us
indicated      that the bonus Kas determined             in the followin;
manner,       DjnalectronVs       performance      score for the total      con-
tract    period     was in the high-good         range with an overall
numerical      average of 88,7,           On the basis of the quarterly
 scores making up this           average,    Dynalectron    was entitled     to
 an award fee of about $39,990,              or about 65 percent       of the
available      amount.      It was decided,        however,   that Dynalectron
 should receive        88.7 percent       of the a-ward fee a;Tailable      Zor
 the total     contract     period,      or about $53,216.       The  board  con-
curred     in the committeess          recommendation,

       The official          determination       of the award fee for the
fourth     quarter      stated     that a review        of performance         for the
entire     1st contract         year revealed        that certain        factors       which
affected      the contractorss           performance        were not completely
within     the control        of the contractor.              It concluded       by stat-
ing that,       in recognition          of such factors          and in consider-
ation    of overall         performance,       an additional         fee of $14,226
was awarded to t'ne contractor.                  As examples of the problems
that were beyond the contractor"s                    control,      it cited      inade-
quate hangar space for aircraft                  maintenance,          nonavailability
of critical        aircraft       parts   due to Department            of Defense re-
quirements,        and the age of certain              aircraft      which made the
contractor's        task more difficult            than would normally             be ex-
pected.       NO specifics         were given in support             of these conten-
tions.

        To have received        an award fee of $53,216 under the es-
tablished     fee distribution       guide,   Dynalectron    would have
needed an average performance            score of about 94-5 for ,the
contract    year,    T'nerefore,     we conclude     that Dynalectron    re-
ceived    an award fee for excellent          performance    when, in fact,
its scores indicated          that its performance       was only high-good.

CHANGES IN RECOMMENDED SCORES NGT
SUPPGRTED BP FACTUAL INFGRHATION

       We found that the board on two occasions             had increased
and on one occasion        had decreased     the committee's     recommended
scores but that the contract          files    contained   no factual   in-
formation    explaining     the board!s     changes.     The board changed
the committee's       recommended scores in three consecutive
evaluation      periods     from   November     1, 1967,     through     July   31,
1968.

         The score changes for the evaluation                  period      November 1,
1967, through         January     28, 1968, resulted         in an increase          of
about $1,480 in the award fee to Dynalectron.                          The    board
had raised      the committee          scores for three of the performance
criteria:       (1) schedule         performance,      (2) personnel,           and (3)
cost control,          The result       was an increase        in the overall
score from 89,7 to 90,6;               but the contract        files     contained
no explanation         of the board's         basis for the change.              Simi-
larly,     score changes by the board in the other two periods
were not supported           by the presentation          of factual        informa-
tion;     however,      these changes had no effect               on the award fee
recommended by the committee                for these two periods.               During
discussions       with us, MSC officials            stated     that,     although
it was difficult           to fully     document all facts           considered      in
a decision,        the reasons for committee             and board actions
should always be fully              documented.

        Our review of the performance               evaluation     records      showed
 that the evaluators         did not retain         documentation       supporting
 their   presentations       and recommendations            to the committee,
 except for the operational            indicators       listed    on page 8.
The committee        chairman    told us that to develop             a file     of
documents      to support     performance         scores would be pointless
because the board did not necessarily                   agree with the award
fee suggested        by the committee.           One evaluator       advised     us
that there was no requirement               that he keep his evaluation
documents     and that he only maintained               such information         until
the end of each reporting             period.       He did, however,        retain
records     of the operational         statistics       which were developed
each month,        The Chief,     Flight      Research Projects        Office,
told us that he destroyed            his evaluation          notes after      the
committee     meetings.
                                    CHAPTER 4

                    CONCLUSIONS AND RECOMMENDATIONS

CONCLUSIONS

       In our opinion,        the performance     evaluation    plans for
the aircraft        maintenance   contracts    have not met the require-
ments of NASA's CPAF contracting            guide,     We believe       that the
fees paid to the contractor           were not adequately       justified
because of the lack of standards            for measurement       of the per-
formance     criteria.

      We believe       also that the establishment      of, and adherence
to,  appropriate       performance  standards   against    which to mea-
sure contractor        performance  would help to preclude

       --contractor      performance     ratings     that      appear    inconsis-
          tent with     objective    output    indicators,

       --evaluation     plan deviations which result   in the award
          of higher   fees that are not related   to higher  levels
          of performance,    and

       --changes   of recommended performance     scores which                   are
          not supported   by documented   factual  information                   at-
          testing  to the reasons   for the changes,

       We believe       that more objective         measurements        could be
made of the contractor's            performance       if steps were taken to
 (1) select      performance     evaluation     criteria       directly      related
to the work output          expected     from the contractor,            (2) estab-
lish   objective       standards    for measurement         of contractor         per-
formance,      and (3) establish         a system to produce,           gather,      and
report    information       on the contractor's          actual     versus     its ex-
pected performance.

RECOMMENDATIONS TO THE NASA ADMINISTRATOR

        We recommend that NASA examine into the requirements
of the aircraft      maintenance      contract    with a view to estab-
lishing    a performance     evaluation      plan which will     place
greater    emphasis on objective        performance    standards     in terms


                                            17
               Y-i   L     1   .aL-




of output criteria      sr.2 that it make use of available      perfor-
mance indicators      such as those discussed    above.    In the event
that NASA concludes       that such standards   cannot be developed,
we believe    that  contracting    for aircraft  maintenance    on a
cost-plus-  award-fee     basis should be reconsidered.



         Y&C ofr'icials       with whom we discussed       this matter        ex-
pressed     a willingness          to explore  the possibilities          of work
order evaluation:;           based on output     measurement       standards.
They noterj:, howe:~rer , that such evaluations               represent      a sig-
nificant     change from the present           evaluation       plan and that
the present        stafiir.2     levels   may not be adequate         to implement
changed evaluation            procedures,




                                          1%
                              CHAPTW5

                          SCOPEOF REVIEW

       We reviewed the policies,     practices,     and procedures
followed by the Manned Spacecraft        Center, Houston, Texas, in
evaluating    the performance of its aircraft         maintenance con-
tractor.     Our review was directed     primarily     toward determin-
ing whether MSC had established       a performance evaluation
plan as set forth by NASA's CPAF contracting             guide and
whether MSC’s implementation      of the performance evaluation
plan provided reasonable assurance that the fee awards were
commensurate with the contractor's        demonstrated performance.
Although our work was performed principally            at MSC, we also
held discussions     with NASA Headquarters      officials.




U.S. GAO Wash.,   D.C.

                                   19