oversight

Analysis of the Proposed Survey by the Civil Aeronautics Board of Claims Against Air Carriers for Cargo Lost or Damaged

Published by the Government Accountability Office on 1971-12-20.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

(.   I   ‘. REPORT TO THL
            OF THE SELECT CbGMI?TEE           (j fo WJ
            ON SMALL 3USINESS                             /
            UNITEW STATES SENATE                    (‘, lj’ j
                                    ;j- “/(,Id’ 1 .-d
                                    d




                   -.-




            BY THE COMPTROLLER GENERAL
         .. OF THE UNITED iSTATES
                 .
                                                                                                                ..--   -



                                               COMPTROLLER      GENERAL?JF          THE       UNiTED   STATES
                                                              WASHINGTON.    D.C.         20548




                  B-142142
             Ld   Dear Mr.        Chairman:

           I               We have examined               the plans for the Civil _.,,_         Aeronautics
                                                                                                  ” ,,“.,“l*,,ll”^*~.l
                                                                                                                  ,.fi            !r
        c d/0     Board’s         proposed         survey of carrier           claim=!         you requested.in
        ’.        y=F      letter         of August 5, 1971, a copy of which is appended to
    I             this     report.           As you requested,            we concentrated           our examina-
        -.        tion on whether                the survey was constructed                  in accordance               with
                  recognized            scientific        and statistical           practices       and whether
                  it therefore             would constitute           an accurate          and true sample
                  which would provide                  the data needed to make a reasonable                               judg-
                  ment as to the adequacy of the present                             release      rate liability
                  of 50 cents a pound.                                                                                   : --.
                                                                                         __.
                                                                                           -
                                                                                         KS*
                           We have been advised                 by a representative             of the Bureau
                  of Economics             of the Civil         Aeronautics         Board that there                  is no
                  written         plan on how the survey data will                       be used and no writ-
                  ten statement              of the decision          criteria       to be used in reaching
                  statistically              reliable       conclusions        on the data obtained                     in the
                  survey.           In view of the lack of written                     information            regarding
                  the overall             survey plan,        our examination            was based primarily
                  on verbal           information         supplied      to us by the Civil               Aeronautics
                  Board representative-and                    details      on the sampling             plan con-
                  tained        in the Civil           Aeronautics        Board’s      Prehearing           Conference
                  Report of August 17, 1971.

                        The survey plan calls      for two samples:        first,     a histor-
    n.            ical  sample for the 6-month period          January   1 to June 30, 1971,
                  and second,    a prospective    sample for the 2-month period            Sep-
. r*              tember 1 to October       31, 1971.     As discussed    below neither        of
                  these samples    is entirely    satisfactory      from a ‘statistical
                  point  of view.
.
                  PLAN FOR HISTORICAL                  SAMPLE

                         This sample will        co.nsist     of 1,189 claims     selected        from
                  the 6-month period         ended June 30,, 1971, during          which there
                  were approximately         48,200 such claims,         The sampling        plan
                  calls   for all carriers        that maintain       numbered listings           of
                  claims    to select     samples on the basis         of recurring       digits.
                  For example,     selection      of every claim having         a.log-in       number
                  ending in 37 would provide              a l-percent   sample.      Northwest


                                                        50TH ANNIVERSARY                    1921-1971
                       T
                               .   ._                                _
              ,                                                                  I
                                                                            .-       ---_          -.._   -   -




                  B-142142


                  Air Lines does not maintain              such a listing    but will  select
                  every 25th claim and thereby              obtain  a 4-percent   sample.
                          Samples for three other airlines                that do not maintain
        ,
        -.        numbered listings           of claims received        will   be selected    ran-
                  domly by Civil.Aeronautics              Board-personnel.          The sample is
        .         disproportionately           assigned     to individual      carriers;   for in-
         -‘       stance,     Eastern Airlines          having a universe       of 6,000 claims
                  and Frontier       Airlines      having a universe         of 600 claims are each
                  assigned      a sample of 60 claims.            Samples for individual         car-
                  riers    range from 25 for Air West to 96 for Pan American Air-
                  lines,

                  OUR VIEWS ON STATISTICAL            VALIDITY      OF --        -;;;.
                  HISTORICAL PLAN

                          The sampling plan used is statistically                   acceptable     for
                  selecting        random samples and the sample size is adequate for
                  providing         statistically       reliable     estimates    at the combined
                  level.       The samples are, however,              too small to provide       statis-
                  tical     validity         for any detailed      analysis    of the data on a
                  carrier-by-carrier              basis and month-by-month         basis.     It is even
                  doubtful       that the estimates            on a,month-by-month       basis without
                  regard to carriers              would be statistically        acceptable.

                        - Obtaining     disproportionate           samples from participating
                  carriers      is not statistically            inaccurate;      individual     carrier
    '             results,      however,    must be properly           weighted     in preparing       sum-
                  mary level       estimates.       If unweighted,          the summary level        esti-
- ..              mates could be severely             distorted.        Also the results        are only
                  applicable       to the 6 months of the year covered by the sample.
                  It will     not be possible         to infer,      with any statistical          vali-
.                 dity,    that carrier       claims for the 6 months of the year not
                  covered have the same characteristics                     as those covered.

                          The Civil  Aeronautics          Board commented               on our findings      in
                  a letter    dated October 18,           1971, a copy of               which is appended
                  to this report.       The Board         has stated  that              it intends    to re-
                  port survey findings       only       in the aggregate                and that proper
                  weighting    would be used in           summarizing   data              from the dispro-
                  portionate     samples obtained          from the various                 carriers.

                                                              2
           .

               B-142142



                       The Board advised      us that it did not agree that results
               from the 6-month historical            sample could not be used in mak-
               ing’inferences       beyond the 6-month period.         The Board further
               stated     that it believed     we had misunderstood      what the data
     .         would cover.       It explained      that the time periods     selected   for
     -.        both historical       and prospective       data were based on the date
     .         that claims     were closed      (paid or-denied)     and not on the date
      -.       that the shipments        were made or claims- filed.

                      Hence a February        1971 claims     settlement     might possibly
               cover a February      1971     shipment    but more likely        a shipment   made
               many months before.          Consequently-the         Board believes     that the
               data provided     for the      6-month historical         and 2-month prospec-
               tive   periods   will   be   fully   representative;Iof       claims. on ship:
               ments made throughout          the year and doubts that including
               claims    closed   in the    remaining     4 months would add to the va-
               lidity    of the data:

                        We cannot agree with the Board’s               view on this point.
               Statistical          theory   is very clear    that inferences        cannot be
               made for any part of a universe              that did not have a chance of
               being included           in t-he sample.    Therefore,       even though the
               claims      included       in the 6-month   sample may have originated            at
               various       times,     the universe    is based on the date the claims
               were paid or denied and not the origination                    dates.     Conse -
               quently       it would be.statistically          invalid     to make inferences
               to the part of the year not included                  in the sample.

               PLAN FOR THE PROSPECTIVE           SAMPLE

                      The sampling     plan for the prospective     sample requires
               each cayrier      to submit   data for the first    500 claims  to be
               processed     in each of the 2 months,     September   and October   1971.
               It is-estimated      that the sample size will      be 10,264 claims   from
               a total    universe    of 16,064.
I.
,
               OUR VIEWS OF VALIDITY         OF PROSPECTIVE       PLAN

                       The sample method is statistically         unsound because it is
               not random.     -The six largest  carriers      process     more than 500
                claims   a month and therefore   the claims       these carriers      pro-
               .cess in excess of 500 a month--a         total   estimated     at 2,900
                  B-142142



                  claims    a month or 5,800 claims,for        the Z-month period--are
                  given no chance of being included          in the sample.       With a
                  sample size this       large,   however, the problem       of unscientific
                  sampling    could be overcome by demonstrating           that the estimated
             -.   5,800 claims     that would be excluded        from the sample are es-
                  sentially    no different     from those included      in the sample.
         i
             -.
                        We have found no indication,    however,   that the Board
                  plans  to make such a demonstration.      Even if such a demonstra-
                  tion were made, the results     would only be applicable   to the
                  2 months from which the sample was selected.

                         As in the case of the histori.cal-              samp-le, complex weight-
                  ing of the results        will     be necessary      to orffset   the effects   of
                  the,disproportionate           method of sample selection.            Even more
                  complex weighting       will     be necessary      if prospective      sample data
                  are combined with       the historical        sample data.        And, even if
                  properly      combined,   there is still        the problem     that 4 months
                  are not represented.

                             In commenting    on our views on the             prospective       sample,
                  the       Civil Aeronautics    Board stated:

                            “GAO objects      to the prospective         data requested           be-
                            cause it is not random.            However,     GAO admits         that
                        -   the data are so extensive            as to be representative.
    ’ -’                    The fact    is that the Bureau does not desire                   to
                            have a random sample for the future                period,      but
                            wants as near 100 percent            as possible.          Taking
I u:                        the first     500 claims     for each carrier          for each
                            month will     cover all of the claims            settled      by most
                            of the carriers.       zain,       these are claims          that are
                            closed    and thus will      represent     claims      filed     in
j    l




i                           months other      than September       and October.”
s

                           Our views on the prospective             sample do not imply or
                  state     that the sample is so large             as to be representative         but
                   rather-that       the problem     of omitting        5,800   or 34 percent     of the
                  claims       for the Z-month period        could be overcome by demonstrat-
                  ing that claims         omitted    are essentially          no different    from
                  those included        in the sample.          Notwithstanding         size, the sam-
                  ple was not selected            on a scientific         basis  and, if the results
                             ,’                              4            ‘,
                 I
                          --

                                                               -.        --          -.--    -




              B-142142



              are to be statistically           valid,   it must be demonstrated             that
              the claims   not included         in the sample are essentially               no dif-
              ferent   from those that         were included.

                     Our previous comment on inferences        beyond the period
         .    covered   by the sample is equally    applicable      here.
         -.
                                                 .-
     l
              OVERALL EVALUATION
         -.
                     In    its letter      to us, the Board stated         its opinion     that
              the data       base was fully      representative      and adequate      for deci-
              sion of      the case and that it would strike              a reasonable    balance
              between      the needs of the proceeding          and the burden of time and
              expense      required     to -produce the data.-       The Board pointed        out
              that it      was only one party         to the investi-&tion       and that it
              did not      unilaterally      determine     what data would be provided           in
              response       to its request.

                      All the other parties      have a right     to be heard,      and the
              examiner      makes the final  ruling.       Consequently    the Board
              stated     that it did not intend        to submit  any further     requests,
              that the case was proceeding           in normal fashion,      and that the
              historical      data were-to  be supplied      on or before     November 1.

                     We agree that the sample is a very large                one and that the
              data submitted       in accordance    with the Board’s         plans may provide
              a reasonable      basis  for judging      the adequacy of the current          re-
              lease rate liability        of 50 cents a pound.          However,    the sam-
              pling    method employed     does not provide      statistical       assurance
              that the sample will        be representative      of all claims        in the
    . -.      universe     because not all claims         had a chance of being selected.

                      In this respect        claims      paid or denied        in 4 months of the
l             year and an estimated            5,800     claims     processed     by major carriers
              during    September     and October           1971 had no chance of being se-
              lected.      Therefore,      from a      statis’tical      viewpoint,     the results
              of this    sample are subject            to challenge         on the basis   that the
              results    could be different            if,     for some reason or another--
              such as seasonal        variation--         the claims      that had no chance of
              being included       are different            from those included       in the sam-
              pie.

                                                           5
                                                               _
                                                          -    -        -    ---     ---   -




             B-142142



                     We believe     that the statistical          deficiency    in the sample
             could be overcome without            great difficulty         by demonstrating   on
             a test basis       that the claims       that had no chance to be selected
             are not significantly         different       from those claims       that were
         .   included     in the sample.        We believe      also that it would be ad-
        *
             visable    for the Board to verify            the accuracy      of the data sub-
                                                         .-
             mitted    by the carriers.



                    We plan to make no further         distribution    of this report
             unless    copies    are specifically    requested,     and then we shall
             make distribution        only after   your agreement     has been obtained
             or public     announcement      has been made-by zou concerning-the-
             contents     of this   report.

                                                      Sincerely             yours,




                                      -..             Comptroller   General
                                                      of the United   States

             The Honorable        Alan Bible
             Chairman,      Select,Committee     on
                Small Business                                5'[. bi a se9 f. ir
    .        United    States     Senate

    .


l




                                                      6
                         L’
                                                --
               c

                                                                                -   -         -     ---       -_-   -

                                                                                                              APPENDIX       I




                              JOSEPH     I..   WARD.                        WASHINGTON.      D.C.    20510
                     ASSIsrANT         GENERAL         CCUNSP.
                                                                                    August          5, 1971



      *            The Honorable        Elmer B. Staats      --
      .I           Comptroller      General    of the United    States
                   General     Accounting     Office
                   Washington,      D. C. 20548

                   Dear Kr.                     Comptroller      General:

                                   This Committee               for the past two-and                  one-half       years
                 has been engaged in a study and hearings                                    on cargo theft            from
                 different         modes of transportation                       and its impact            on small
                 businessmen.              In this       regard,        in P;iovember of 1969, I filed                      a
                  statement        with the Civil              Aeronautics             Board expressing             concern
                  for the air carr.iers                liability           limitation           and its effect           on
                 ,cgyt ._..  security           I  also      introduced            legislation,           S. 1763,       which
                                           7-
        _- -. 4  would       statutorsly          set air carrier                liability         at actual        cash
         t, 1 'I value.         As you will-note                from the enclosed                statement       I made at
         9 ' the time of filing                    this       legislation,             this b2.11 is consistent
         "$      with the recommendations                       of the Senate Small Business                        Committee
                  in its report            on air cargo theft.
         , \
       ., *                        The Civil         Aeronautics             Board is at the present                   conducting
    ..
      1           an investigation              of air carrier               liability          and claims       rules
    ,            practices.            In this       investigation,                the Board has requested                  all
    +,            certificated           air carriers             to provide           it with       certain      information
                 necessary          to resolve         the serious             questions         of the adequacy            of
l
                  liability         limitations          on air cargo.                 The air carriers             have
                 refused        to supply         all the information                    requested        by the Board
                  and have sought relief                   by request.ing              permission         to file      a limited,
                  random sample survey.

                                I would' appreciate     your office       examining      from a
                   technical    standpoint   this   proposed       survey and advising       me if it
                   is constructed     by recognized     scientific       and statistical      practices,
                   in such a manner as to provide          an accurate       and true random



                                                                            1
                        s
                            _-
             I
                                                                           --

         1                                                                          APPENDIX I


                 The Honorable  Elmer          B, S-hats
                 August 5, 1971
                 Page 2


                 sample which would provide              the necessary   data needed to
                 make a reasonable     judgment          as to the adequacy of the present
    d            50 cents per pound release              rate liability.     Your assistance
    A.           to me and the Senate Small              Business Committee in this regard
                 is -deeply appreciated.
l
                                                                      _.
    *.


                                 With   all   best   wishes,




                 Encl




                                                               2
               i           --                                             -.
           .
                                                                     -.                -   ---      --.-       -


                                                                                                      APPENDIX IT

                                               CIVIL     AERONAUTlCS                  BOARD
                                                       WASHINGTON,         D.C.   20428            INREPLYREFERTO:        B-8-64

                                                             OCT 18 1971




      *                Mr. D. L. Scantlebury
     * .
                       Director
                       Division   of Financial and
 l

     *.
                          General Management Studies                              _.
                       U. S. General Accounting    Office
                       Washington, D. C. 20548

                       Dear Mr. Scantlebury:

                              This is in reply to your           letter      to the Chairman dated
                       September 15, 1971, requesting              comment-on yoyr draft report pn          --
                       an examination   of a survey of           air-carrier       claims being undertaken
                       by the parties   to the Liability             and Claims Rules and Practices
                       Investigation,   Docket 19923.            You state that your examination         was
                       undertaken at the request of             the Chairman, Select Committee on
                       Small Business, United States              Senate.

                               Since this proceeding      is a formal investigation     which has
                       been assigned for hearing before an examiner, the Board has not
                       passed upon such matters as the adequacy of the evidence,              which in
                       the normal course would only come before the Board upon review of
                       the examiner's      decision.     The Bureau of Economics is a party to the
                       proceeding     and has participated       extensively  in the prehearing    con-
                       ferences that have been held with respect to the evidence to be
                   -   furnished     by carriers     and shippers in response to the Bureau's
                       requests.      We therefore     asked the Bureau to review your draft
 .                     repart,     and we enclose a memorandum setting        forth their comments.

                             In the event that your report is publicly    released                         in final
,'                     form, it would be appreciated   if you would include this                           letter   and
                       the Bureau's comments in any such final    report.




                                                                                   Cjngressional   Relation&         -"

                       Enclosure



                                                                     3
                                                                                 --     --                  -._-        .-

                                                                                                            APPENDIX                II



                                           CIVIL     AERONAUTICS            BOARD

                                                                   October        18,        1971


        MEMORANDUM

        TO:              Directar,         Office      of   Community         and     Congressional                Relations
    *
        FROM:            Director,         Bureau      of Economics

.       SUBJECT :        Letter       of   September         15,    1971,     enclosing
                                                                              _^                    draft   report           from        GAO


                 The subject            draft       report       deals with an opinion              by GAO of the
        sampling         techniques           established           by the parties          and approved       by the
        examiner         to develop           statistical           data on freight           loss and damage claims
        in the Liability                Rules case, Docket                19923.       We shall      comment on the
        points      in the GAO report                   seriatim,               _      __.                                 -:
                                                                                       -?
                 Historical           data - GAO finds                the historical          data to be supplied
        by the carriers               to be statistically                 acceptable        in the aggregate           but
        states      that       such data are too small                    to be statisticaliy            valid     on a
        carrier-by-carrier                  or month-by-month               basis.       Since the Bureau intends
        that     the data will              be analyzed           in the aggregate,             the overall      results
        should      be statistically                  valid,        The rules       under investigation            are
        generally          uniform        for all         the carriers;          and, as is the normal             case in
        an industrywide               rate investigation,                 the lawfulness           of the rules        will
        be judged          for the-most             part on evidence             produced       by the entire        industry.
        In our judgment,                the historical              data which will           be provided      in response                     to
        the Bureau’s             request       are fully          adequate       for the purpose         intended;          i.e.,
        to ascertain            carrier        and shipper            experience       under the rules         and to
        evaluate        their       lawfulness.

                 GAO also suggests      that    the data supplied      by individual                                   carriers
        should      be properly   weighted      in estimating    industry    results.                                   The Bureau
        fully     intends    to use appropriate       weightings    in summarizing                                   the data.

                With respect        to GAO’s concern       that data for six months will             not
        be representative           of a full  year,     GAO has apparently        misunderstood         what
        the data will        cover.     The time periods        selected     for both historical          and
        prospective       data are based on the date claims               were closed      (paid    or
        denied),     and not on the date the shipments                were made or claims        filed.
        Hence,     a February       1971 claim   settlement      might    possibly    cover a February
        1971 shipment        but more likely      a shipment       made many months before.
                     &                                                                       _.
                                     _-
                 .                                                                                                             ..--         -
                                                                                       .*.
             7




         J                                                                                                                            APPENDIX        II


                                Consequently,         we believe       that the data provided        for the six-month
                                historical       and two-month         prospective    periods   will    be fully   representa-
                                tive     of claims      on shipments       made throughout    the year.        We seriously
                                doubt that       including      claims     closed   in the remaining      four months would
                                add to the validity           of the data.

                                        Prospective      data - GAO objects           to the prospective            data requested
                         ’      because      it is not random.            However,    GAO admits      that     the data are so
                                extensive       as to be representative.              The fact      is that      the Bureau    does
                                not des,ire      to have a random sample for the future                    period,      but wants as
                                near 100 percent         as possible.          Taking    the first      500 claims        for each
                                carrier      for each month will           cover all     of the claimssettled              by most
f-                              of the carriers.          Again,       these areTaims          that are closed          and thus
     .                          will    represent    claims      filed     in months     other    than September         and
                                October.

                                          Overall      comments
                                                                               [See GAO note                below.]
                                                                                            The bases--.on which the claims                     data
                               are to be developed                 have been the subject                  of%xtensive           discussion       and
                               exploration           by all      the parties          to the investigation                 in prehearing
                               conferences           presided        over by an experienced                   hearing      examiner.       It is
                               the Bureau’s            intention         that the data submitted                   be broadly        based and
                               representative              of carrier        and shipper          claims       experience.           The parties
                               agreed       that     the development            of large         quantities         of data on the basis
                               of analyses           of past        (closed)      claims      would be extremely                burdensome       and
                               expensive         and that        data developed            on a day-to-day              future      basis  could
                               be more readily              supplied.         The Bureau would not accept                       placing    full
                               reliance        on a prospective              survey,       which would delay               resolution      of the
                               case for many months.                     We also felt          that a future            survey alone would
                               not be as reliable                as such a.survey              plus representative                past data.
                               Therefore,         it was ruled             by the examiner            that     the data would be pro-
                               vided      for both past and future                    periods,        with that         for the past on a
                             - relatively          limited       but adequate           sample,       and that        for the future         on a
                               mOre comprehensive                basis.       In our opinion,               the data base is fully
                               representative              and adequate         for decision            of the case and strikes                 a
                               reasonable         balance        between the needs of the proceeding                            and the burden
                               of time and expense                 required       to produce          the data.
L’
                                       We should   also           point      out as a practical           matter    that the Bureau,
                                as one party     to the           investigation,           does not unilaterally           determine        what
t
                                data will    be provided             in response         to its requests.           All  the other         parties
                                have a right     to be          heard,       and the examiner          makes the final        ruling.          In
                                any event,    for the           reasons        indicated      previously,        we do not intend          to
                                submit   any further            requests.           The case is proceeding            in normal       fashion,
                                and the historical              data are to be supplied                 on or before     November        1.




                         .     GAO note:            ‘Deleted  material  refers                 to matters     in.a     draft           of   this   report
                                                     which are not included                  in the final       report.

                                                                                      5