(. I ‘. REPORT TO THL OF THE SELECT CbGMI?TEE (j fo WJ ON SMALL 3USINESS / UNITEW STATES SENATE (‘, lj’ j ;j- “/(,Id’ 1 .-d d -.- BY THE COMPTROLLER GENERAL .. OF THE UNITED iSTATES . ..-- - COMPTROLLER GENERAL?JF THE UNiTED STATES WASHINGTON. D.C. 20548 B-142142 Ld Dear Mr. Chairman: I We have examined the plans for the Civil _.,,_ Aeronautics ” ,,“.,“l*,,ll”^*~.l ,.fi !r c d/0 Board’s proposed survey of carrier claim=! you requested.in ’. y=F letter of August 5, 1971, a copy of which is appended to I this report. As you requested, we concentrated our examina- -. tion on whether the survey was constructed in accordance with recognized scientific and statistical practices and whether it therefore would constitute an accurate and true sample which would provide the data needed to make a reasonable judg- ment as to the adequacy of the present release rate liability of 50 cents a pound. : --. __. - KS* We have been advised by a representative of the Bureau of Economics of the Civil Aeronautics Board that there is no written plan on how the survey data will be used and no writ- ten statement of the decision criteria to be used in reaching statistically reliable conclusions on the data obtained in the survey. In view of the lack of written information regarding the overall survey plan, our examination was based primarily on verbal information supplied to us by the Civil Aeronautics Board representative-and details on the sampling plan con- tained in the Civil Aeronautics Board’s Prehearing Conference Report of August 17, 1971. The survey plan calls for two samples: first, a histor- n. ical sample for the 6-month period January 1 to June 30, 1971, and second, a prospective sample for the 2-month period Sep- . r* tember 1 to October 31, 1971. As discussed below neither of these samples is entirely satisfactory from a ‘statistical point of view. . PLAN FOR HISTORICAL SAMPLE This sample will co.nsist of 1,189 claims selected from the 6-month period ended June 30,, 1971, during which there were approximately 48,200 such claims, The sampling plan calls for all carriers that maintain numbered listings of claims to select samples on the basis of recurring digits. For example, selection of every claim having a.log-in number ending in 37 would provide a l-percent sample. Northwest 50TH ANNIVERSARY 1921-1971 T . ._ _ , I .- ---_ -.._ - - B-142142 Air Lines does not maintain such a listing but will select every 25th claim and thereby obtain a 4-percent sample. Samples for three other airlines that do not maintain , -. numbered listings of claims received will be selected ran- domly by Civil.Aeronautics Board-personnel. The sample is . disproportionately assigned to individual carriers; for in- -‘ stance, Eastern Airlines having a universe of 6,000 claims and Frontier Airlines having a universe of 600 claims are each assigned a sample of 60 claims. Samples for individual car- riers range from 25 for Air West to 96 for Pan American Air- lines, OUR VIEWS ON STATISTICAL VALIDITY OF -- -;;;. HISTORICAL PLAN The sampling plan used is statistically acceptable for selecting random samples and the sample size is adequate for providing statistically reliable estimates at the combined level. The samples are, however, too small to provide statis- tical validity for any detailed analysis of the data on a carrier-by-carrier basis and month-by-month basis. It is even doubtful that the estimates on a,month-by-month basis without regard to carriers would be statistically acceptable. - Obtaining disproportionate samples from participating carriers is not statistically inaccurate; individual carrier ' results, however, must be properly weighted in preparing sum- mary level estimates. If unweighted, the summary level esti- - .. mates could be severely distorted. Also the results are only applicable to the 6 months of the year covered by the sample. It will not be possible to infer, with any statistical vali- . dity, that carrier claims for the 6 months of the year not covered have the same characteristics as those covered. The Civil Aeronautics Board commented on our findings in a letter dated October 18, 1971, a copy of which is appended to this report. The Board has stated that it intends to re- port survey findings only in the aggregate and that proper weighting would be used in summarizing data from the dispro- portionate samples obtained from the various carriers. 2 . B-142142 The Board advised us that it did not agree that results from the 6-month historical sample could not be used in mak- ing’inferences beyond the 6-month period. The Board further stated that it believed we had misunderstood what the data . would cover. It explained that the time periods selected for -. both historical and prospective data were based on the date . that claims were closed (paid or-denied) and not on the date -. that the shipments were made or claims- filed. Hence a February 1971 claims settlement might possibly cover a February 1971 shipment but more likely a shipment made many months before. Consequently-the Board believes that the data provided for the 6-month historical and 2-month prospec- tive periods will be fully representative;Iof claims. on ship: ments made throughout the year and doubts that including claims closed in the remaining 4 months would add to the va- lidity of the data: We cannot agree with the Board’s view on this point. Statistical theory is very clear that inferences cannot be made for any part of a universe that did not have a chance of being included in t-he sample. Therefore, even though the claims included in the 6-month sample may have originated at various times, the universe is based on the date the claims were paid or denied and not the origination dates. Conse - quently it would be.statistically invalid to make inferences to the part of the year not included in the sample. PLAN FOR THE PROSPECTIVE SAMPLE The sampling plan for the prospective sample requires each cayrier to submit data for the first 500 claims to be processed in each of the 2 months, September and October 1971. It is-estimated that the sample size will be 10,264 claims from a total universe of 16,064. I. , OUR VIEWS OF VALIDITY OF PROSPECTIVE PLAN The sample method is statistically unsound because it is not random. -The six largest carriers process more than 500 claims a month and therefore the claims these carriers pro- .cess in excess of 500 a month--a total estimated at 2,900 B-142142 claims a month or 5,800 claims,for the Z-month period--are given no chance of being included in the sample. With a sample size this large, however, the problem of unscientific sampling could be overcome by demonstrating that the estimated -. 5,800 claims that would be excluded from the sample are es- sentially no different from those included in the sample. i -. We have found no indication, however, that the Board plans to make such a demonstration. Even if such a demonstra- tion were made, the results would only be applicable to the 2 months from which the sample was selected. As in the case of the histori.cal- samp-le, complex weight- ing of the results will be necessary to orffset the effects of the,disproportionate method of sample selection. Even more complex weighting will be necessary if prospective sample data are combined with the historical sample data. And, even if properly combined, there is still the problem that 4 months are not represented. In commenting on our views on the prospective sample, the Civil Aeronautics Board stated: “GAO objects to the prospective data requested be- cause it is not random. However, GAO admits that - the data are so extensive as to be representative. ’ -’ The fact is that the Bureau does not desire to have a random sample for the future period, but wants as near 100 percent as possible. Taking I u: the first 500 claims for each carrier for each month will cover all of the claims settled by most of the carriers. zain, these are claims that are closed and thus will represent claims filed in j l i months other than September and October.” s Our views on the prospective sample do not imply or state that the sample is so large as to be representative but rather-that the problem of omitting 5,800 or 34 percent of the claims for the Z-month period could be overcome by demonstrat- ing that claims omitted are essentially no different from those included in the sample. Notwithstanding size, the sam- ple was not selected on a scientific basis and, if the results ,’ 4 ‘, I -- -. -- -.-- - B-142142 are to be statistically valid, it must be demonstrated that the claims not included in the sample are essentially no dif- ferent from those that were included. Our previous comment on inferences beyond the period . covered by the sample is equally applicable here. -. .- l OVERALL EVALUATION -. In its letter to us, the Board stated its opinion that the data base was fully representative and adequate for deci- sion of the case and that it would strike a reasonable balance between the needs of the proceeding and the burden of time and expense required to -produce the data.- The Board pointed out that it was only one party to the investi-&tion and that it did not unilaterally determine what data would be provided in response to its request. All the other parties have a right to be heard, and the examiner makes the final ruling. Consequently the Board stated that it did not intend to submit any further requests, that the case was proceeding in normal fashion, and that the historical data were-to be supplied on or before November 1. We agree that the sample is a very large one and that the data submitted in accordance with the Board’s plans may provide a reasonable basis for judging the adequacy of the current re- lease rate liability of 50 cents a pound. However, the sam- pling method employed does not provide statistical assurance that the sample will be representative of all claims in the . -. universe because not all claims had a chance of being selected. In this respect claims paid or denied in 4 months of the l year and an estimated 5,800 claims processed by major carriers during September and October 1971 had no chance of being se- lected. Therefore, from a statis’tical viewpoint, the results of this sample are subject to challenge on the basis that the results could be different if, for some reason or another-- such as seasonal variation-- the claims that had no chance of being included are different from those included in the sam- pie. 5 _ - - - --- --- - B-142142 We believe that the statistical deficiency in the sample could be overcome without great difficulty by demonstrating on a test basis that the claims that had no chance to be selected are not significantly different from those claims that were . included in the sample. We believe also that it would be ad- * visable for the Board to verify the accuracy of the data sub- .- mitted by the carriers. We plan to make no further distribution of this report unless copies are specifically requested, and then we shall make distribution only after your agreement has been obtained or public announcement has been made-by zou concerning-the- contents of this report. Sincerely yours, -.. Comptroller General of the United States The Honorable Alan Bible Chairman, Select,Committee on Small Business 5'[. bi a se9 f. ir . United States Senate . l 6 L’ -- c - - - --- -_- - APPENDIX I JOSEPH I.. WARD. WASHINGTON. D.C. 20510 ASSIsrANT GENERAL CCUNSP. August 5, 1971 * The Honorable Elmer B. Staats -- .I Comptroller General of the United States General Accounting Office Washington, D. C. 20548 Dear Kr. Comptroller General: This Committee for the past two-and one-half years has been engaged in a study and hearings on cargo theft from different modes of transportation and its impact on small businessmen. In this regard, in P;iovember of 1969, I filed a statement with the Civil Aeronautics Board expressing concern for the air carr.iers liability limitation and its effect on ,cgyt ._.. security I also introduced legislation, S. 1763, which 7- _- -. 4 would statutorsly set air carrier liability at actual cash t, 1 'I value. As you will-note from the enclosed statement I made at 9 ' the time of filing this legislation, this b2.11 is consistent "$ with the recommendations of the Senate Small Business Committee in its report on air cargo theft. , \ ., * The Civil Aeronautics Board is at the present conducting .. 1 an investigation of air carrier liability and claims rules , practices. In this investigation, the Board has requested all +, certificated air carriers to provide it with certain information necessary to resolve the serious questions of the adequacy of l liability limitations on air cargo. The air carriers have refused to supply all the information requested by the Board and have sought relief by request.ing permission to file a limited, random sample survey. I would' appreciate your office examining from a technical standpoint this proposed survey and advising me if it is constructed by recognized scientific and statistical practices, in such a manner as to provide an accurate and true random 1 s _- I -- 1 APPENDIX I The Honorable Elmer B, S-hats August 5, 1971 Page 2 sample which would provide the necessary data needed to make a reasonable judgment as to the adequacy of the present d 50 cents per pound release rate liability. Your assistance A. to me and the Senate Small Business Committee in this regard is -deeply appreciated. l _. *. With all best wishes, Encl 2 i -- -. . -. - --- --.- - APPENDIX IT CIVIL AERONAUTlCS BOARD WASHINGTON, D.C. 20428 INREPLYREFERTO: B-8-64 OCT 18 1971 * Mr. D. L. Scantlebury * . Director Division of Financial and l *. General Management Studies _. U. S. General Accounting Office Washington, D. C. 20548 Dear Mr. Scantlebury: This is in reply to your letter to the Chairman dated September 15, 1971, requesting comment-on yoyr draft report pn -- an examination of a survey of air-carrier claims being undertaken by the parties to the Liability and Claims Rules and Practices Investigation, Docket 19923. You state that your examination was undertaken at the request of the Chairman, Select Committee on Small Business, United States Senate. Since this proceeding is a formal investigation which has been assigned for hearing before an examiner, the Board has not passed upon such matters as the adequacy of the evidence, which in the normal course would only come before the Board upon review of the examiner's decision. The Bureau of Economics is a party to the proceeding and has participated extensively in the prehearing con- ferences that have been held with respect to the evidence to be - furnished by carriers and shippers in response to the Bureau's requests. We therefore asked the Bureau to review your draft . repart, and we enclose a memorandum setting forth their comments. In the event that your report is publicly released in final ,' form, it would be appreciated if you would include this letter and the Bureau's comments in any such final report. Cjngressional Relation& -" Enclosure 3 -- -- -._- .- APPENDIX II CIVIL AERONAUTICS BOARD October 18, 1971 MEMORANDUM TO: Directar, Office of Community and Congressional Relations * FROM: Director, Bureau of Economics . SUBJECT : Letter of September 15, 1971, enclosing _^ draft report from GAO The subject draft report deals with an opinion by GAO of the sampling techniques established by the parties and approved by the examiner to develop statistical data on freight loss and damage claims in the Liability Rules case, Docket 19923. We shall comment on the points in the GAO report seriatim, _ __. -: -? Historical data - GAO finds the historical data to be supplied by the carriers to be statistically acceptable in the aggregate but states that such data are too small to be statisticaliy valid on a carrier-by-carrier or month-by-month basis. Since the Bureau intends that the data will be analyzed in the aggregate, the overall results should be statistically valid, The rules under investigation are generally uniform for all the carriers; and, as is the normal case in an industrywide rate investigation, the lawfulness of the rules will be judged for the-most part on evidence produced by the entire industry. In our judgment, the historical data which will be provided in response to the Bureau’s request are fully adequate for the purpose intended; i.e., to ascertain carrier and shipper experience under the rules and to evaluate their lawfulness. GAO also suggests that the data supplied by individual carriers should be properly weighted in estimating industry results. The Bureau fully intends to use appropriate weightings in summarizing the data. With respect to GAO’s concern that data for six months will not be representative of a full year, GAO has apparently misunderstood what the data will cover. The time periods selected for both historical and prospective data are based on the date claims were closed (paid or denied), and not on the date the shipments were made or claims filed. Hence, a February 1971 claim settlement might possibly cover a February 1971 shipment but more likely a shipment made many months before. & _. _- . ..-- - .*. 7 J APPENDIX II Consequently, we believe that the data provided for the six-month historical and two-month prospective periods will be fully representa- tive of claims on shipments made throughout the year. We seriously doubt that including claims closed in the remaining four months would add to the validity of the data. Prospective data - GAO objects to the prospective data requested ’ because it is not random. However, GAO admits that the data are so extensive as to be representative. The fact is that the Bureau does not des,ire to have a random sample for the future period, but wants as near 100 percent as possible. Taking the first 500 claims for each carrier for each month will cover all of the claimssettled by most f- of the carriers. Again, these areTaims that are closed and thus . will represent claims filed in months other than September and October. Overall comments [See GAO note below.] The bases--.on which the claims data are to be developed have been the subject of%xtensive discussion and exploration by all the parties to the investigation in prehearing conferences presided over by an experienced hearing examiner. It is the Bureau’s intention that the data submitted be broadly based and representative of carrier and shipper claims experience. The parties agreed that the development of large quantities of data on the basis of analyses of past (closed) claims would be extremely burdensome and expensive and that data developed on a day-to-day future basis could be more readily supplied. The Bureau would not accept placing full reliance on a prospective survey, which would delay resolution of the case for many months. We also felt that a future survey alone would not be as reliable as such a.survey plus representative past data. Therefore, it was ruled by the examiner that the data would be pro- vided for both past and future periods, with that for the past on a - relatively limited but adequate sample, and that for the future on a mOre comprehensive basis. In our opinion, the data base is fully representative and adequate for decision of the case and strikes a reasonable balance between the needs of the proceeding and the burden of time and expense required to produce the data. L’ We should also point out as a practical matter that the Bureau, as one party to the investigation, does not unilaterally determine what t data will be provided in response to its requests. All the other parties have a right to be heard, and the examiner makes the final ruling. In any event, for the reasons indicated previously, we do not intend to submit any further requests. The case is proceeding in normal fashion, and the historical data are to be supplied on or before November 1. . GAO note: ‘Deleted material refers to matters in.a draft of this report which are not included in the final report. 5
Analysis of the Proposed Survey by the Civil Aeronautics Board of Claims Against Air Carriers for Cargo Lost or Damaged
Published by the Government Accountability Office on 1971-12-20.
Below is a raw (and likely hideous) rendition of the original report. (PDF)