United States General Accounting Office GAO Report to Congressional Requesters November 1997 HUMAN FACTORS FAA’s Guidance and Oversight of Pilot Crew Resource Management Training Can Be Improved GAO/RCED-98-7 United States GAO General Accounting Office Washington, D.C. 20548 Resources, Community, and Economic Development Division B-275381 November 24, 1997 The Honorable John McCain Chairman, Committee on Commerce, Science, and Transportation United States Senate The Honorable Slade Gorton Chairman, Subcommittee on Aviation Committee on Commerce, Science, and Transportation United States Senate Previous studies of aviation safety have found that pilot performance is a major contributor to airline accidents and incidents (events that affect or could affect a flight’s safety). Therefore, training to improve pilots’ performance has been a primary effort to improve airline safety. As part of this effort, some airlines have provided training in crew resource management (CRM) since the early 1980s, and the Federal Aviation Administration (FAA) will require all airlines to have implemented this training for pilots by March 1998. CRM is an approach to improving pilot performance that focuses on better coordination—among members of the cockpit crew as well as among the cockpit crew and flight attendants, dispatchers, and air traffic controllers—to handle certain routine and emergency situations. Airlines can meet the CRM training requirement in one of two ways: (1) by following FAA’s traditional requirements for training pilots and crew—specified in part 121 of the federal aviation regulations1—or (2) by instituting the Advanced Qualification Program (AQP),2 which combines CRM training with technical training for pilots. Part 121 training requirements have been in place without significant modification since the 1970s, and until 1990, all airlines had to meet these requirements. Since 1990, FAA has offered airlines AQP training as an alternative to traditional part 121 training, and eight major airlines have chosen to train their pilots under AQP requirements. This report responds to your request that we examine the role of airline pilots’ performance in accidents and FAA’s efforts to address any inadequate performance. Specifically, we agreed to address the following: 1 14 C.F.R. part 121, subparts N and O. 2 Special Federal Aviation Regulation No. 58—Advanced Qualification Program. Page 1 GAO/RCED-98-7 Human Factors B-275381 (1) What are the types and frequency of accidents in which an airline pilot’s performance was cited as a contributing factor, including those in which failure to use CRM principles was identified, and (2) how adequate is FAA’s guidance for and oversight of the airlines’ implementation of pilots’ training for CRM? We limited our review to the accidents and incidents experienced and training implemented by the 10 major U.S. airlines—those generating $1 billion or more in revenues annually.3 Of the 169 accidents that involved the major airlines and that were Results in Brief investigated and reported on in detail by the National Transportation Safety Board from 1983 through 1995, about 30 percent were caused in part by the pilots’ performance, according to our analysis. In at least one-third of these accidents (about 15), we determined that the pilots did not correctly use the principles of crew resource management. For example, according to the National Transportation Safety Board, just before the 1994 crash in Charlotte, North Carolina, which killed 37 people, the aircraft had encountered a sudden change in wind direction and the captain gave an incorrect order to the first officer, who did not question the order, as crew resource management principles would require. Furthermore, during the same period, of the nearly 4,000 incidents, we found that about one-fifth were caused in part by the pilots’ performance. FAA’s guidance for and oversight of training in crew resource management does not ensure the adequacy of this training under part 121, while they do under the new Advanced Qualification Program. FAA’s guidance for the implementation of the Advanced Qualification Program specifies a process for curriculum development that the airlines must follow in order to integrate training in crew resource management with technical flying skills.4 FAA inspectors overseeing this training assess the curriculum to see if FAA’s process has been followed; this assessment also enables them to determine whether the pilots’ training under this curriculum is adequate. In contrast, although FAA requires airlines to teach crew resource management in their traditional part 121 training, the guidance it provides on how to develop the curriculum for this training is ambiguous and does not provide standards that inspectors can use to evaluate airlines’ training in crew resource management. Because the Advanced Qualification Program training generally differs from traditional part 121 training in how 3 These airlines are Alaska, American, America West, Continental, Delta, Northwest, Southwest, Trans World, United, and US Airways. 4 Initially, crew resource management was known as “cockpit resource management” and referred only to individuals on the flight deck—that is, to pilots and flight engineers. Page 2 GAO/RCED-98-7 Human Factors B-275381 it develops a curriculum for training in crew resource management, the guidance for this training in the Advanced Qualification Program may not be applicable to training for crew resource management under part 121. Therefore, FAA needs to develop guidance for teaching crew resource management under traditional part 121 training. Furthermore, although 8 of the 10 major airlines plan to train all their pilots under AQP, the need for guidance on crew resource management training under part 121 remains—both for those airlines that have opted not to enter the Advanced Qualification Program as well as for those that participate in the program but will nonetheless continue to have some of their pilots trained under part 121 for up to 8 years as they make the transition to the Advanced Qualification Program. Airline travel is one of the safest modes of public transportation in the Background United States. The current level of airline safety has been achieved, in part, because the airline industry and government regulatory agencies have implemented rigorous pilot training and evaluation programs. The major airlines have training programs for pilots that focus on, among other things, maintaining flying skills, qualifying to fly new types of aircraft, and acquiring skills in dealing with emergencies. FAA’soriginal regulations for the airlines’ general training programs—referred to in this report as part 121—spell out the number of hours of training required in particular areas, such as the time spent practicing emergency procedures. Effective for 1996, FAA instituted a requirement for CRM training under part 121 that states the following: “After March 19, 1998, no certificate holder [airline] may use a person as a flight crewmember, and after March 19, 1999, no certificate holder may use a person as a flight attendant or aircraft dispatcher unless that person has completed approved crew resource management or dispatcher resource management initial training, as applicable, with that certificate holder or with another certificate holder.”5 FAA believes that this training should improve flight crews’ performance.6 5 14 C.F.R., section 121.404. 6 This requirement applies to all airlines operating under part 121 and those airlines certified under part 135 that conduct training under part 121. Airlines now operating under part 121 use aircraft configured for 10 or more passengers. New rules adopted by FAA in 1995 require certain commuter operators conducting scheduled operations under part 135 to conduct those operations under part 121 beginning in March 1997. Included were those airlines conducting scheduled operations carrying passengers with aircraft configured for 10 to 30 seats. Page 3 GAO/RCED-98-7 Human Factors B-275381 As an alternative to training under these regulations, airlines may apply to participate in AQP.7 Eight of the 10 major airlines have applied to, and been approved for participation in, AQP. Unlike traditional part 121 training, AQP specifies the criteria for the required level of performance in certain types of maneuvers, rather than hours of training, and it integrates CRM training with technical flying skills. The airlines are expected to fully implement AQP over a period of time, up to 8 years. Full implementation means that the airlines have trained their pilots for each type of aircraft they fly. Training, however, occurs only after the airline has gone through three other stages: (1) getting approval to participate in the program, (2) developing a training curriculum, and (3) training instructors. Continuing crew training, the last stage, is to occur annually. Responsibility for AQP and traditional part 121 training rests with different FAA branches. The AQP Branch within the Office of Flight Standards Services oversees AQP, and the Branch expects to transfer many of its oversight responsibilities to inspectors in the field as each airline fully implements its AQP. The administration of traditional part 121 training is divided between the Air Carrier Training Branch, which sets training requirements, and the flight standards inspectors in the field, who are responsible for overseeing the training. FAA’s inspectors periodically review and approve airlines’ curricula and training materials and observe training. CRM is a “human factors” approach for improving aviation safety by preventing or managing pilots’ errors. Human factors refers to a multidisciplinary effort to develop information about human capabilities and limitations and to apply this information to equipment, systems, facilities, procedures, jobs, environments, training, staffing, and personnel management for safe and effective human performance. Under this approach, pilots are trained to recognize potential mistakes in judgment or actions and to compensate for them to prevent accidents and incidents.8 7 FAA issued the final Special Federal Aviation Regulation (SFAR) 58 for AQP on October 2, 1990 and the termination date for the regulation has been extended to October 2, 2000. The SFAR 58 is found in part 121. The AQP advisory circular was issued on August 9, 1991. 8 The National Transportation Safety Board (NTSB), the official source of information on airline accidents, defines accidents as occurrences in which individuals are killed or suffer serious injury, or the aircraft is substantially damaged. By NTSB’s definition, accidents can range from fatal crashes in which all on board are killed to events in which only one person suffers a broken bone and the aircraft is not damaged, to still others in which there is substantial aircraft damage but no fatalities or serious injuries. NTSB generally distinguishes between accidents and incidents. NTSB defines incidents as occurrences other than accidents associated with the operation of an aircraft that affect or could affect the safety of operations. (49 C.F.R. section 830.2) Page 4 GAO/RCED-98-7 Human Factors B-275381 For example, in training for initial departure, CRM training has the captain practice briefing the crew about the actions to be taken if the takeoff must be aborted because of an emergency. CRM also teaches the crew to question orders when they believe they have information that indicates these orders are inappropriate. Similarly, CRM training teaches the crew to anticipate problems and make decisions that take these anticipated problems into account. About 30 percent of the 169 accidents and 18 percent of the 3,901 incidents Airline Pilots’ that occurred from 1983 through 1995 were caused at least in part by Performance Was a pilots’ performance, according to our analysis of the National Contributing Factor in Transportation Safety Board’s (NTSB) and FAA’s data. Furthermore, the accident data indicate that nearly one-third of the accidents occurred Many Accidents and because the pilots either did not follow, or did not correctly follow, CRM Incidents principles. The most frequently occurring accidents and incidents included collisions on the ground with objects and other airplanes, flights through turbulent weather that resulted in injuries, and deviations from flight paths that had the potential to cause an in-flight collision. Data Show Pilots’ On the ground, pilot performance was associated most frequently with Performance Contributed airplanes colliding with vehicles, buildings, other equipment, or animals. to Accidents and Incidents This was the case for both accidents (32 percent) and incidents (34 percent). Figure 1 shows the types of accidents and incidents on the ground reported from 1983 through 1995. Page 5 GAO/RCED-98-7 Human Factors B-275381 Figure 1: On-the-Ground Accidents and Incidents Associated With Pilots’ Performance b Accidents a Incidents On ground On ground collision with collision with object object 29% Other 34% 32% 44% Other 6% 15% Ran off runway 18% 10% 7% 5% Landing gear Hard landing collapsed Loss of control on ground Loss of control Ran off on ground runway a NTSB cited 62 events associated with pilots’ performance in 169 accidents. b FAA cited 446 events associated with pilots’ performance in 3,901 incident reports. Sources: FAA and NTSB. In the air, pilot performance was most frequently associated with injuries to passengers and flight attendants during turbulent weather—41 percent of accidents and 12 percent of incidents. Figure 2 shows the types of accidents and incidents in the air that were reported. Page 6 GAO/RCED-98-7 Human Factors B-275381 Figure 2: In-the-Air Accidents and Incidents Associated With Pilots’ Performance Accidents a Incidents b Jet exhaust blast 1% Loss of control in flight 1% Engine malfunction 3% Other Struck ground object (i.e., antenna) Aborted 6% Encountered turbulence 15% take-off in flight 4% 12% Encountered turbulence in 41% Landing gear flight 9% not lowered 76% 31% Other Loss of control in flight Note: Amounts may not add to 100 percent due to rounding. a NTSB cited 46 events associated with pilots’ performance in 169 accidents. b FAA cited 209 events associated with pilots’ performance in 3,901 incidents. Sources: FAA and NTSB. In addition to the accidents and incidents discussed above, FAA maintains data separately for those occasions on which pilots failed to comply with the air traffic controller’s instructions—such as not staying on the directed flight path and/or entering a runway without clearance.9 Of the 1,471 unauthorized maneuvers from 1987 through 1995,10 80 percent occurred in the air, and most of these (73 percent) occurred when pilots did not maintain their assigned altitude levels. The unauthorized pilot maneuvers on the ground were most often (69 percent) associated with pilots’ moving airplanes onto runways without authorization from the air traffic control 9 These data are found in FAA’s pilot deviation database. 10 These data were available from FAA’s National Aviation Safety Data Analysis Center only for these years. Page 7 GAO/RCED-98-7 Human Factors B-275381 tower. These types of incidents have the potential to cause accidents. For example, the December 1990 crash at the Detroit Metropolitan Airport occurred when an airplane taxied onto a runway being used for takeoff by another airplane and collided with that airplane. Twelve people died. The first plane had not gotten permission from the control tower to enter this runway, as it should have. Figure 3 shows the most frequently reported unauthorized pilot maneuvers in the air and on the ground. Figure 3: Most Frequently Reported Unauthorized Pilot Maneuvers in the Air and on the Ground a b In-the-air maneuvers On-the-ground maneuvers Take-off on the wrong Other 4% runway/taxiway 3% Landing without clearance Course clearance deviation Land on the wrong 4% runway/airport 5% Other 23% 9% 10% Take-off without clearance 73% 69% Altitude clearance violation Enter taxi/runway without clearance a FAA reported 1,110 unauthorized pilot maneuvers in the air. b FAA reported 258 unauthorized pilot maneuvers on the ground. Source: FAA’s pilot deviation database. Deficiencies in CRM In our analysis of accidents, we found deficiencies in the airline pilots’ use Contributed to Accidents of CRM in nearly one-third of all accidents involving pilots’ performance. Moreover, we found CRM deficiencies in half of the serious accidents in Page 8 GAO/RCED-98-7 Human Factors B-275381 which there was at least one fatality. About 46 percent of these CRM deficiencies involved a lack of coordination among members of the cockpit crew, as well as the captain’s failure to assign tasks to other crew members and to effectively supervise the crew. Generally, these CRM deficiencies illustrated the importance of effective communication. For example, in the Charlotte, North Carolina, crash in July 1994, communication among crew members did not occur, according to NTSB’s accident investigation report. NTSB believes that the captain, who was not flying the aircraft at the time and could not see the ground because of poor visibility, became disoriented and commanded the first officer, “down, push it down,” even though they were encountering windshear, which is a sudden change in wind direction. The first officer did not question the order, as he should have, according to NTSB, because the windshear was creating an unstable situation; the plane could not recover from the sudden downward shift in direction caused by following the captain’s order. The plane crashed nose down into the ground, and 37 people died. Similarly, in a June 1984 accident in Detroit, Michigan, a lack of communication between the crew and air traffic controllers during a landing in a severe thunderstorm contributed to the accident, according to the NTSB report. The crew did not request clarification about the weather conditions or change its course of action to take these conditions into account. The winds associated with the storm forced the plane down precipitously, causing an emergency landing without the landing gear’s being fully extended. The plane skidded off the runway, causing serious damage to the aircraft and an emergency evacuation of the passengers. NTSB reported that the lack of CRM practices was a probable cause of the accident. The National Aeronautics and Space Administration reported similar results in its analysis of pilot reports submitted to its voluntary reporting system.11 Nearly half of the reports cited deficiencies in the pilots’ use of CRM principles; about 53 percent of the CRM deficiencies concerned coordination among members, assignment of tasks, and crew supervision. 11 An Analysis of part 121 Crew Resource Management Incidents, National Aeronautics and Space Administration, Quick Response No. 296, Aviation Safety Reporting System (Feb. 6, 1997). Page 9 GAO/RCED-98-7 Human Factors B-275381 For AQP training, FAA has specified the process airlines need to follow to FAA’s Guidance and develop and implement a curriculum that integrates CRM concepts with Oversight Do Not technical flying skills, but FAA’s guidance for CRM training under part 121 Ensure Effectiveness does not have the same degree of specificity. As a result, inspectors overseeing training under part 121 do not have standards they can use to of CRM Training evaluate airlines’ CRM training curriculum and the delivery of that training. Generally, inspectors could not use the guidance provided under AQP to evaluate part 121 training for the CRM curriculum because the curricula developed under the two programs differ significantly. As a result, airlines continue to need specific guidance for CRM under part 121—both those airlines that have opted not to enter AQP as well those that will continue to train at least some of their crews under part 121 until they have fully implemented AQP, which could take up to 8 years. FAA’s Guidance for CRM Once an airline elects to participate in AQP, it must follow SFAR 58 (the AQP Training Is Detailed Under regulation) for developing a formal curriculum—including assessing the AQP but Not Under Part skills pilots need to safely operate the aircraft they fly, developing curriculum objectives for teaching those skills, having measurable criteria 121 for evaluating whether the pilots have achieved those objectives, and developing materials to teach those objectives. FAA must approve this curriculum. Furthermore, AQP requires all airlines to train their pilots in simulators so that they gain experience with a number of emergency situations. Finally, airlines must submit data to FAA demonstrating that their crews have mastered the skills they need to fly for those airlines.12 In developing its AQP curriculum, an airline is required to integrate CRM training into every aspect of its crews’ training. As a result, the pilots trained under AQP are assessed on CRM principles as well as on technical flying skills. For example, when a pilot changes the aircraft’s altitude—a technical flying skill—CRM principles dictate that this pilot inform the other pilot by verbally announcing the new altitude while continually pointing to the altitude indicator until the other pilot also points to the altitude indicator and repeats the new altitude. This procedure is used to ensure that neither pilot will fail to maintain the appropriate altitude. In contrast, FAA’s requirements for CRM training under part 121 do not require airlines to develop a curriculum for CRM training with measurable criteria or to integrate that curriculum with other aspects of part 121 training. For the CRM curriculum under part 121, FAA provides suggested 12 Training records for pilots are maintained under part 121. Performance data for crews submitted to FAA under AQP permit the agency to conduct its own analyses of crews’ mastery. Page 10 GAO/RCED-98-7 Human Factors B-275381 training topics but does not clearly lay out how the airlines are to introduce these topics into their training programs, according to airline officials and FAA inspectors. For example, FAA recommends that airlines train crews in “workload management and situational awareness.” For this training, FAA suggests such topics as “preparation/planning/vigilance” and “workload distribution/distraction avoidance.” However, for those airlines that choose to integrate these topics with technical flying skills, FAA does not explain how the airlines are to do so.13 The lack of specificity in FAA’s guidance for the development of a CRM curriculum under part 121 contrasts with the detailed guidance FAA provides for the development of a curriculum on technical flying skills. For example, FAA’s guidance on how pilots are to respond to windshear under part 121 directs them in a number of technical flying skills, such as how to handle the rudder, but it is silent on how to employ CRM principles in this situation. In contrast, under AQP, FAA’s guidance instructs the airlines to specify not only the technical skills but also the CRM principles that must be applied in a windshear situation. Because FAA’s guidance on CRM training under part 121 is less specific, airlines vary in how they deliver their CRM training. While all the airlines provide classroom training in CRM principles under part 121 training, they may not integrate this training with technical flying skills. For example, airlines may (1) train pilots in technical flying skills in flight simulators without integrating CRM principles or (2) integrate CRM principles with technical flying skills in flight simulators. Generally, we found that CRM training had been integrated with technical flight training to a higher degree at those airlines that were in later phases of AQP implementation. FAA’s Oversight of CRM In developing AQP, FAA incorporated procedures for evaluating CRM training Training Is Adequate for and developed a process for ensuring that FAA inspectors would have the AQP but Not for Part 121 criteria they need to conduct the evaluations for pilots’ training on different types of aircraft. Specifically, AQP provides a systematic way of identifying the tasks and subtasks involved in a particular phase of flight. Therefore, an inspector observing the training program can determine whether CRM principles are being invoked in a given flight situation. For example, when a crew is preparing for landing, AQP specifies that the first officer, if unsure of the planned course of action in the event of a missed approach, is to ask the captain to clarify the plan so that both have a full understanding of the actions they will take. Similarly, if a flight has to be 13 See FAA’s Advisory Circular 120-51B, Crew Resource Management Training. Page 11 GAO/RCED-98-7 Human Factors B-275381 diverted from one airport to another, the captain is to direct the first officer to (1) get out the maps for the alternate airport, (2) notify the flight attendants, and (3) make the announcement to the passengers. This delegation of tasks allows the captain time to handle radio contact with the airline’s dispatchers and air traffic controllers, obtain weather updates at the alternate airport, and fly the plane. In the early stages of AQP implementation, the AQP Branch is evaluating airlines’ training. FAA will transfer this responsibility to inspectors in the field as airlines fully implement AQP. Field inspectors will be trained in evaluating the CRM training as an integral part of their evaluation of AQP training. The inspectors at those airlines that had progressed beyond the initial phases of AQP noted that they had received AQP training at the airlines for which they were responsible. Moreover, all of the inspectors we spoke with maintained that while certain facets of AQP were fixed, some parts were still evolving. As a result of the program’s flexibility and evolution, the inspectors pointed out that it was not possible to structure a training program for them that could cover every aspect of AQP at every airline. Despite this fluidity, these inspectors said that the AQP Branch Office made sure that the program’s standards were maintained across airlines. While the evaluation of the delivery of CRM training is incorporated into the oversight process for AQP training, it is not under traditional part 121 training. Moreover, FAA has not provided its inspectors with any specific guidance or training for evaluating airlines’ CRM training under part 121. Although FAA inspectors may obtain some CRM training from a 3-hour computerized interactive course, this lack of guidance for evaluating CRM training under part 121 is troublesome to the inspectors we spoke with because of what they view as an inherent conflict between performance expectations for individuals under part 121 and crew performance expectations articulated in CRM principles.14 Under part 121, pilots are to master technical flying skills and perform these skills without reliance on any other crew member. In contrast, CRM principles and training teach pilots how to use to maximum effect the abilities and experience of other crew members, as well as their own technical flying skills. Without formal FAA instructions, inspectors have developed their own approaches to this evaluation. For example, one inspector said that he based his approval on his belief that the airline for which he was 14 In a previous review of FAA’s training for its inspector workforce—Aviation Safety: Targeting and Training of FAA’s Safety Inspector Workforce (GAO/T-RCED-96-26, Apr. 30, 1996)—we found that some inspectors were unaware that needed training was available through computer-based courses. Page 12 GAO/RCED-98-7 Human Factors B-275381 responsible “had a good safety record” and “would probably establish a good program.” Another inspector said that in approving any training program, he sought guidance first from any applicable federal aviation regulation; the Inspector’s Handbook, applicable advisory circulars; and, finally, any other FAA publication, such as the Introduction to CRM Training. However, this inspector added that these sources did not provide the criteria he needed to evaluate CRM training. As a result, he looked for behaviors such as crew members’ “working together” to resolve problems, “catching errors,” or “dealing with the consequences resulting from uncaught errors.” These ad hoc approaches to evaluating the delivery of CRM training are not satisfactory to FAA officials at headquarters or to officials for at least one airline. FAA officials told us that the agency needed additional CRM training for its inspectors conducting reviews under part 121. In addition, officials at one airline told us that the lack of specific guidance and training for FAA inspectors responsible for evaluating CRM training under part 121 has hampered FAA’s ability to review CRM programs. Furthermore, the problems FAA inspectors face in evaluating CRM training under part 121 will continue indefinitely in the absence of clearer guidance from FAA for those airlines that have decided not to enter AQP and for those airlines in the program that have not fully implemented it. Because AQP is implemented by the type of aircraft the crew flies, even the airlines that have been accepted for AQP will continue to provide some CRM training under part 121. For the eight airlines implementing AQP, we estimate that only about one-third of their crews have begun to receive AQP training. Therefore, most crews are still receiving traditional training under part 121, and some will continue do so for up to 8 years. As of September 1997, the airlines’ estimated dates for completing the transition to AQP training ranged between 2000 and 2005. (See table 1.) Page 13 GAO/RCED-98-7 Human Factors B-275381 Table 1: Estimate of Completed AQP Implementation by Airline Pilots being trained under AQP as of Estimated year of Airline Year begun September 1997 completion Alaska 1994 0 2002 American 1994 0 2005 Continental 1997 0 2002 Delta 1992 60% 2001 Northwest 1994 23% 2001 Trans World 1995 0 2000 United 1991 50% 2001 US Airwaysa 1994 0 2000 a Estimate is for one aircraft type. US Airways’ aircraft fleet composition is under review, and no determination has been made to place any other aircraft types into AQP until the aircraft fleet plan is resolved. For the flying public, safety is the paramount issue, and FAA and the Conclusions airlines have worked to provide rigorous training programs for pilots. Crew resource management, which focuses on making the best use of all available experience and skills in the cockpit, is increasingly seen as an important component of safe flights. FAA recognized the importance of crew resource management by requiring all airlines to include training in these principles and by incorporating crew resource management into its Advanced Qualification Program. Pilots’ performance is not the only factor in airline accidents, but it is an important one. We identified pilots’ performance as the cause of about one-third of all the accidents and nearly one-fifth of the incidents for the 10 major airlines from 1983 through 1995. Training for safer performance by pilots that teaches crew resource management can occur under either the Advanced Qualification Program or part 121. However, while FAA’s guidance for the implementation of the Advanced Qualification Program specifies a process for curriculum development that integrates this training with training in technical flying skills, FAA’s guidance for curriculum development under part 121 is ambiguous and does not provide standards that inspectors can use to evaluate and approve airlines’ training in crew resource management. As a result, FAA cannot be assured that airlines are developing a curriculum for teaching crew resource management that will effectively teach pilots how Page 14 GAO/RCED-98-7 Human Factors B-275381 to best use all the skills and experience available to them in the cockpit. Furthermore, without specificity in the development of training for crew resource management under part 121 and without any guidance on how to evaluate this training under part 121, FAA inspectors are relying on their own experience in observing pilots or even on the belief that the airline “would probably establish a good program.” These problems are especially troublesome because pilots who have not completed FAA-approved crew resource management training by March 1998 may not fly for airlines. To help ensure that airlines appropriately train pilots in CRM principles Recommendations under part 121 and that FAA inspectors are able to uniformly evaluate this CRM training, we recommend that the Secretary of Transportation direct the Administrator of FAA to develop a process that airlines must follow for creating a CRM curriculum, with measurable criteria, under part 121 as it has for the Advanced Qualification Program. We provided a draft of this report to FAA for review and comment. We met Agency Comments with the Deputy Associate Administrator for Regulation and Certification, and Our Evaluation the Deputy Director of Flight Standards Services, the Managers for the Air Carrier Training Branch and the Advanced Qualification Program, and other officials. FAA commended our review of CRM training at the nation’s airlines. FAA accepted the report’s recommendation in part. FAA agreed that it should ensure that pilots are appropriately trained and noted that CRM training can provide desirable consequences in aviation safety. It further agreed that uniform evaluation of CRM training using measurable criteria is a commendable objective. However, FAA stated that science has not yet developed valid, reliable criteria for measuring CRM performance. FAA also agreed that more can be done to develop a process that airlines and inspectors can follow to create a CRM curriculum. FAA indicated that better guidance would be provided in a number of ways, such as updating Advisory Circular 120-51, Crew Resource Management Training, and supplemental guidance for inspectors included in the inspectors’ handbook and holding regional meetings with CRM specialists from Flight Standards Services and other organizations. We concur with FAA that CRM training for pilots could improve aviation safety. However, we believe that before the contribution of CRM training to aviation safety can be measured, it is necessary to determine the extent to which the delivery of CRM training for pilots has occurred. We further concur with FAA that more should be done to develop processes for Page 15 GAO/RCED-98-7 Human Factors B-275381 airlines and inspectors to follow in creating a CRM curriculum. We believe that until FAA establishes a process for CRM curriculum development that includes an assessment of the extent to which pilots have mastered that curriculum, it will not be possible to measure CRM’s performance in contributing to aviation safety. To determine the extent to which inadequate performance by pilots was a Scope and problem for the 10 major U.S. airlines, we examined the types and Methodology frequency of safety-threatening events—incidents and accidents—from 1983 through 1995. To determine the adequacy of FAA’ s guidance for and oversight of pilots’ training, we reviewed FAA’s role in the airlines’ implementation of CRM. We focused primarily on CRM training because FAA has described the failure to apply CRM principles as a more important contributing factor in accidents than technical flying skills. We also compared FAA’s rules and regulations and other guidance for CRM training with that provided for other training programs, as well as interviewed FAA and airline officials. A detailed discussion of our methodology is presented in appendix I. Related GAO products are listed at the end of this report. Our work was performed from October 1996 through October 1997 in accordance with generally accepted government auditing standards. As arranged with your offices, unless you publicly announce its contents earlier, we plan no further distribution of this report until 30 days after the date of this letter. At that time, we will provide copies of the report to the Secretary of Transportation, appropriate congressional committees, and other interested parties. We will also make copies available to others upon request. Page 16 GAO/RCED-98-7 Human Factors B-275381 Please contact me at (202) 512-3650 if you or your staff have any questions about this report. Major contributors to this report are listed in appendix II. Gerald L. Dillingham Associate Director, Transportation Issues Page 17 GAO/RCED-98-7 Human Factors Contents Letter 1 Appendix I 20 Methodology Appendix II 23 Major Contributors to This Report Related GAO Products 24 Tables Table 1: Estimate of Completed AQP Implementation by Airline 14 Table I.1: Classification Framework for Categorizing CRM Issues 21 Figures Figure 1: On-the-Ground Accidents and Incidents Associated 6 With Pilots’ Performance Figure 2: In-the-Air Accidents and Incidents Associated With 7 Pilots’ Performance Figure 3: Most Frequently Reported Unauthorized Pilot 8 Maneuvers in the Air and on the Ground Abbreviations AQP Advanced Qualification Program ASRS Aviation Safety Reporting System CRM crew resource management FAA Federal Aviation Administration GAO General Accounting Office NASA National Aeronautics and Space Administration NTSB National Transportation Safety Board Page 18 GAO/RCED-98-7 Human Factors Page 19 GAO/RCED-98-7 Human Factors Appendix I Methodology To identify the types and frequencies of accidents and incidents—safety-threatening events—related to pilot performance, we reviewed accident and incident data, including pilot deviations, contained in the National Transportation Board’s (NTSB) and the Federal Aviation Administration’s (FAA) electronic databases. We obtained these data from FAA’s National Aviation Safety Data Analysis Center. We limited our review to the reported events in accident and incident data sources involving the 10 major U.S. passenger airlines from 1983 through 1995. We did not independently verify these data. To facilitate the comparison of accidents with incidents in our analysis of the types and frequencies of safety-threatening events, we made two adjustments to the data. First, because of differences in the way information is recorded in these databases, we matched the similar categories contained in both databases and used these categories in our analysis. For example, both NTSB’s and FAA’s databases contain the category “on ground collision with object,” which means an airplane struck an object, such as a vehicle or structure, while moving on the ground. Second, because the occurrences of events in accidents closely conform to those in incidents, we used the events that occurred in each of the 169 accidents as our unit of analysis. In our analysis of crew resource management (CRM) deficiencies, we used the accident as the unit of analysis because NTSB’s findings of CRM deficiencies were by accident and not by the individual events that occurred within accidents. To characterize the prevalence of pilot performance as a factor in safety-threatening events over time and between airlines, we examined FAA’s incident and pilot deviation databases. We used these two databases because they are the only such sources with adequate numbers of observations to make such comparisons. To determine the extent to which the inadequate use of CRM by pilots contributed to accidents and incidents, we performed a content analysis of the textual information found in the factual reports, briefs, and final reports of the 169 accidents investigated by NTSB from 1983 through 1995. We then classified CRM deficiencies according to the classification framework presented at a National Aeronautics and Space Administration (NASA)/Ames workshop in 1980.15 This framework groups CRM issues into five broad clusters: 15 Murphy, M.R. (1980). “Analysis of Eighty-four Commercial Aviation Incidents: Implications for a Resource Management Approach to Crew Training.” 1980 Proceedings Annual Reliability and Maintainability Symposium. Ames Research Center, NASA, Moffet Field, California. Page 20 GAO/RCED-98-7 Human Factors Appendix I Methodology • (1) Resource management—the application of specialized cognitive skills to effectively and efficiently utilize available resources, such as the ability to plan, organize, and communicate. • (2) Organization processes—crew members’ actions and behaviors in the context of their assigned duties and expected responsibilities. • (3) Personal factors—the knowledge, skills, abilities, and limitations that individual crew members bring with them to the cockpit. • (4) Material resources internal to the aircraft—the cockpit crew’s appropriate, effective, and efficient use of instructional items, such as checklists, and navigational charts and equipment, such as on-board weather radar, navigational controls, and engine fire extinguisher. • (5) Resources external to the aircraft—those people (air traffic controllers), entities (airports), and circumstances (emerging poor weather) that may affect pilots’ plans, decisions, and actions. Table I.1 shows the classification framework used to categorize CRM issues. Table I.1: Classification Framework for Categorizing CRM Issues Clusters Material resources Resource Organization internal to the Resources external management processes Personal factors aircraft to the aircraft CRM issues •Social •Role •Knowledge •Textual •Human •Communication •Monitoring •Proficiency •Equipment •Facility •Leadership •Workload •Experience •Environment •Management •Motivation •Planning •Stress •Problem-solving reaction •Decision-making •Fatigue To verify the results of our content analysis, we requested a similar analysis by NASA’s Aviation Safety Reporting System (ASRS) staff of voluntarily submitted pilot reports contained in the ASRS database. According to the aviation experts we consulted, ASRS incident reports provide the best source of information on deficiencies in CRM. Furthermore, because ASRS staff are most familiar with the data and have expertise in analyzing this free-form data, we concluded that it was more appropriate for them to perform this analysis. To evaluate the adequacy of FAA’s oversight of airline pilot training, we obtained FAA’s training policies, requirements, guidance, and handbooks Page 21 GAO/RCED-98-7 Human Factors Appendix I Methodology relevant to CRM training. We discussed training programs, including CRM, and training procedures with appropriate FAA officials, including officials in the Office of System Safety, the Office of Regulation and Certification’s Flight Standards Services, the Advanced Qualification Program Branch, the Office of Accident Investigation, and the Human Factors Division. In addition, we discussed airline training evaluation and approval processes and obtained training documents from FAA inspectors responsible for monitoring airline training. Finally, we contacted safety directors and trainers at the major airlines and obtained documents on their policies, procedures, research, and training curricula. We requested comments from recognized experts in the field of human factors in academia and the aviation industry, pilots, and government officials from FAA, NTSB, and NASA. We incorporated their comments where appropriate and made adjustments to our methodology as warranted. Page 22 GAO/RCED-98-7 Human Factors Appendix II Major Contributors to This Report John H. Anderson, Jr. Resources, Emilie G. Heller Community, and Luann M. Moy Economic Richard R. Scott E. Jerry Seigler Development Carol Herrnstadt Shulman Division, Washington, Robert E. White D.C. Mindi G. Weisenbloom Office of General Counsel Christopher A. Keisling Atlanta Regional Office Page 23 GAO/RCED-98-7 Human Factors Related GAO Products Aviation Safety: New Airlines Illustrate Long-Standing Problems in FAA’s Inspection Program (GAO/RCED-97-2, Oct. 17, 1996). Human Factors: Status of Efforts to Integrate Research on Human Factors Into FAA’s Activities (GAO/RCED-96-151, June 27, 1996). Military Aircraft Safety: Significant Improvements Since 1975 (GAO/NSIAD-96-69BR, Feb. 1, 1996). Aviation Safety: Data Problems Threaten FAA Strides on Safety Analysis System (GAO/AIMD-95-27, Feb. 8, 1995). Aviation Safety: Unresolved Issues Involving U.S.-Registered Aircraft (GAO/RCED-93-135, June 18, 1993). Aviation Safety: Changes Needed in FAA’s Service Difficulty Reporting Program (GAO/RCED-91-24, Mar. 21, 1991). (341513) Page 24 GAO/RCED-98-7 Human Factors Ordering Information The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. VISA and MasterCard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent. Orders by mail: U.S. General Accounting Office P.O. Box 37050 Washington, DC 20013 or visit: Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000 or by using fax number (202) 512-6061, or TDD (202) 512-2537. Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone. A recorded menu will provide information on how to obtain these lists. For information on how to access GAO reports on the INTERNET, send an e-mail message with "info" in the body to: firstname.lastname@example.org or visit GAO’s World Wide Web Home Page at: http://www.gao.gov PRINTED ON RECYCLED PAPER United States Bulk Rate General Accounting Office Postage & Fees Paid Washington, D.C. 20548-0001 GAO Permit No. G100 Official Business Penalty for Private Use $300 Address Correction Requested
Human Factors: FAA's Guidance and Oversight of Pilot Crew Resource Management Training Can Be Improved
Published by the Government Accountability Office on 1997-11-24.
Below is a raw (and likely hideous) rendition of the original report. (PDF)