Human Factors: FAA's Guidance and Oversight of Pilot Crew Resource Management Training Can Be Improved

Published by the Government Accountability Office on 1997-11-24.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                United States General Accounting Office

GAO             Report to Congressional Requesters

November 1997
                HUMAN FACTORS
                FAA’s Guidance and
                Oversight of Pilot Crew
                Resource Management
                Training Can Be

      United States
GAO   General Accounting Office
      Washington, D.C. 20548

      Resources, Community, and
      Economic Development Division


      November 24, 1997

      The Honorable John McCain
      Chairman, Committee on Commerce,
        Science, and Transportation
      United States Senate

      The Honorable Slade Gorton
      Chairman, Subcommittee on Aviation
      Committee on Commerce, Science, and
      United States Senate

      Previous studies of aviation safety have found that pilot performance is a
      major contributor to airline accidents and incidents (events that affect or
      could affect a flight’s safety). Therefore, training to improve pilots’
      performance has been a primary effort to improve airline safety. As part of
      this effort, some airlines have provided training in crew resource
      management (CRM) since the early 1980s, and the Federal Aviation
      Administration (FAA) will require all airlines to have implemented this
      training for pilots by March 1998. CRM is an approach to improving pilot
      performance that focuses on better coordination—among members of the
      cockpit crew as well as among the cockpit crew and flight attendants,
      dispatchers, and air traffic controllers—to handle certain routine and
      emergency situations.

      Airlines can meet the CRM training requirement in one of two ways: (1) by
      following FAA’s traditional requirements for training pilots and
      crew—specified in part 121 of the federal aviation regulations1—or (2) by
      instituting the Advanced Qualification Program (AQP),2 which combines
      CRM training with technical training for pilots. Part 121 training
      requirements have been in place without significant modification since the
      1970s, and until 1990, all airlines had to meet these requirements. Since
      1990, FAA has offered airlines AQP training as an alternative to traditional
      part 121 training, and eight major airlines have chosen to train their pilots
      under AQP requirements.

      This report responds to your request that we examine the role of airline
      pilots’ performance in accidents and FAA’s efforts to address any
      inadequate performance. Specifically, we agreed to address the following:

       14 C.F.R. part 121, subparts N and O.
       Special Federal Aviation Regulation No. 58—Advanced Qualification Program.

      Page 1                                                          GAO/RCED-98-7 Human Factors

                   (1) What are the types and frequency of accidents in which an airline
                   pilot’s performance was cited as a contributing factor, including those in
                   which failure to use CRM principles was identified, and (2) how adequate is
                   FAA’s guidance for and oversight of the airlines’ implementation of pilots’
                   training for CRM? We limited our review to the accidents and incidents
                   experienced and training implemented by the 10 major U.S.
                   airlines—those generating $1 billion or more in revenues annually.3

                   Of the 169 accidents that involved the major airlines and that were
Results in Brief   investigated and reported on in detail by the National Transportation
                   Safety Board from 1983 through 1995, about 30 percent were caused in
                   part by the pilots’ performance, according to our analysis. In at least
                   one-third of these accidents (about 15), we determined that the pilots did
                   not correctly use the principles of crew resource management. For
                   example, according to the National Transportation Safety Board, just
                   before the 1994 crash in Charlotte, North Carolina, which killed 37 people,
                   the aircraft had encountered a sudden change in wind direction and the
                   captain gave an incorrect order to the first officer, who did not question
                   the order, as crew resource management principles would require.
                   Furthermore, during the same period, of the nearly 4,000 incidents, we
                   found that about one-fifth were caused in part by the pilots’ performance.

                   FAA’s guidance for and oversight of training in crew resource management
                   does not ensure the adequacy of this training under part 121, while they do
                   under the new Advanced Qualification Program. FAA’s guidance for the
                   implementation of the Advanced Qualification Program specifies a process
                   for curriculum development that the airlines must follow in order to
                   integrate training in crew resource management with technical flying
                   skills.4 FAA inspectors overseeing this training assess the curriculum to see
                   if FAA’s process has been followed; this assessment also enables them to
                   determine whether the pilots’ training under this curriculum is adequate.
                   In contrast, although FAA requires airlines to teach crew resource
                   management in their traditional part 121 training, the guidance it provides
                   on how to develop the curriculum for this training is ambiguous and does
                   not provide standards that inspectors can use to evaluate airlines’ training
                   in crew resource management. Because the Advanced Qualification
                   Program training generally differs from traditional part 121 training in how

                   These airlines are Alaska, American, America West, Continental, Delta, Northwest, Southwest, Trans
                   World, United, and US Airways.
                    Initially, crew resource management was known as “cockpit resource management” and referred only
                   to individuals on the flight deck—that is, to pilots and flight engineers.

                   Page 2                                                           GAO/RCED-98-7 Human Factors

             it develops a curriculum for training in crew resource management, the
             guidance for this training in the Advanced Qualification Program may not
             be applicable to training for crew resource management under part 121.
             Therefore, FAA needs to develop guidance for teaching crew resource
             management under traditional part 121 training. Furthermore, although 8
             of the 10 major airlines plan to train all their pilots under AQP, the need for
             guidance on crew resource management training under part 121
             remains—both for those airlines that have opted not to enter the
             Advanced Qualification Program as well as for those that participate in the
             program but will nonetheless continue to have some of their pilots trained
             under part 121 for up to 8 years as they make the transition to the
             Advanced Qualification Program.

             Airline travel is one of the safest modes of public transportation in the
Background   United States. The current level of airline safety has been achieved, in part,
             because the airline industry and government regulatory agencies have
             implemented rigorous pilot training and evaluation programs. The major
             airlines have training programs for pilots that focus on, among other
             things, maintaining flying skills, qualifying to fly new types of aircraft, and
             acquiring skills in dealing with emergencies.

             FAA’soriginal regulations for the airlines’ general training
             programs—referred to in this report as part 121—spell out the number of
             hours of training required in particular areas, such as the time spent
             practicing emergency procedures. Effective for 1996, FAA instituted a
             requirement for CRM training under part 121 that states the following:

             “After March 19, 1998, no certificate holder [airline] may use a person as a
             flight crewmember, and after March 19, 1999, no certificate holder may use
             a person as a flight attendant or aircraft dispatcher unless that person has
             completed approved crew resource management or dispatcher resource
             management initial training, as applicable, with that certificate holder or
             with another certificate holder.”5 FAA believes that this training should
             improve flight crews’ performance.6

              14 C.F.R., section 121.404.
              This requirement applies to all airlines operating under part 121 and those airlines certified under part
             135 that conduct training under part 121. Airlines now operating under part 121 use aircraft configured
             for 10 or more passengers. New rules adopted by FAA in 1995 require certain commuter operators
             conducting scheduled operations under part 135 to conduct those operations under part 121 beginning
             in March 1997. Included were those airlines conducting scheduled operations carrying passengers with
             aircraft configured for 10 to 30 seats.

             Page 3                                                               GAO/RCED-98-7 Human Factors

As an alternative to training under these regulations, airlines may apply to
participate in AQP.7 Eight of the 10 major airlines have applied to, and been
approved for participation in, AQP. Unlike traditional part 121 training, AQP
specifies the criteria for the required level of performance in certain types
of maneuvers, rather than hours of training, and it integrates CRM training
with technical flying skills. The airlines are expected to fully implement
AQP over a period of time, up to 8 years. Full implementation means that
the airlines have trained their pilots for each type of aircraft they fly.
Training, however, occurs only after the airline has gone through three
other stages: (1) getting approval to participate in the program,
(2) developing a training curriculum, and (3) training instructors.
Continuing crew training, the last stage, is to occur annually.

Responsibility for AQP and traditional part 121 training rests with different
FAA branches. The AQP Branch within the Office of Flight Standards
Services oversees AQP, and the Branch expects to transfer many of its
oversight responsibilities to inspectors in the field as each airline fully
implements its AQP. The administration of traditional part 121 training is
divided between the Air Carrier Training Branch, which sets training
requirements, and the flight standards inspectors in the field, who are
responsible for overseeing the training. FAA’s inspectors periodically
review and approve airlines’ curricula and training materials and observe

CRM  is a “human factors” approach for improving aviation safety by
preventing or managing pilots’ errors. Human factors refers to a
multidisciplinary effort to develop information about human capabilities
and limitations and to apply this information to equipment, systems,
facilities, procedures, jobs, environments, training, staffing, and personnel
management for safe and effective human performance. Under this
approach, pilots are trained to recognize potential mistakes in judgment or
actions and to compensate for them to prevent accidents and incidents.8

 FAA issued the final Special Federal Aviation Regulation (SFAR) 58 for AQP on October 2, 1990 and
the termination date for the regulation has been extended to October 2, 2000. The SFAR 58 is found in
part 121. The AQP advisory circular was issued on August 9, 1991.
 The National Transportation Safety Board (NTSB), the official source of information on airline
accidents, defines accidents as occurrences in which individuals are killed or suffer serious injury, or
the aircraft is substantially damaged. By NTSB’s definition, accidents can range from fatal crashes in
which all on board are killed to events in which only one person suffers a broken bone and the aircraft
is not damaged, to still others in which there is substantial aircraft damage but no fatalities or serious
injuries. NTSB generally distinguishes between accidents and incidents. NTSB defines incidents as
occurrences other than accidents associated with the operation of an aircraft that affect or could
affect the safety of operations. (49 C.F.R. section 830.2)

Page 4                                                                GAO/RCED-98-7 Human Factors

                             For example, in training for initial departure, CRM training has the captain
                             practice briefing the crew about the actions to be taken if the takeoff must
                             be aborted because of an emergency. CRM also teaches the crew to
                             question orders when they believe they have information that indicates
                             these orders are inappropriate. Similarly, CRM training teaches the crew to
                             anticipate problems and make decisions that take these anticipated
                             problems into account.

                             About 30 percent of the 169 accidents and 18 percent of the 3,901 incidents
Airline Pilots’              that occurred from 1983 through 1995 were caused at least in part by
Performance Was a            pilots’ performance, according to our analysis of the National
Contributing Factor in       Transportation Safety Board’s (NTSB) and FAA’s data. Furthermore, the
                             accident data indicate that nearly one-third of the accidents occurred
Many Accidents and           because the pilots either did not follow, or did not correctly follow, CRM
Incidents                    principles. The most frequently occurring accidents and incidents included
                             collisions on the ground with objects and other airplanes, flights through
                             turbulent weather that resulted in injuries, and deviations from flight paths
                             that had the potential to cause an in-flight collision.

Data Show Pilots’            On the ground, pilot performance was associated most frequently with
Performance Contributed      airplanes colliding with vehicles, buildings, other equipment, or animals.
to Accidents and Incidents   This was the case for both accidents (32 percent) and incidents
                             (34 percent). Figure 1 shows the types of accidents and incidents on the
                             ground reported from 1983 through 1995.

                             Page 5                                            GAO/RCED-98-7 Human Factors

Figure 1: On-the-Ground Accidents and Incidents Associated With Pilots’ Performance

                           Accidents a                                                      Incidents

                                                                 On ground
  On ground                                                      collision with
  collision with                                                 object
  object                            29%                Other                          34%
                                                                                                            44%          Other

                         15%                         Ran off runway
                                  18%                                                 10%
                                                                                              7% 5%
                                                                      Landing gear
          Hard landing                                                collapsed
                                     Loss of control
                                     on ground                                       Loss of control   Ran off
                                                                                     on ground         runway

                                              NTSB cited 62 events associated with pilots’ performance in 169 accidents.
                                                  FAA cited 446 events associated with pilots’ performance in 3,901 incident reports.

                                              Sources: FAA and NTSB.

                                              In the air, pilot performance was most frequently associated with injuries
                                              to passengers and flight attendants during turbulent weather—41 percent
                                              of accidents and 12 percent of incidents. Figure 2 shows the types of
                                              accidents and incidents in the air that were reported.

                                              Page 6                                                               GAO/RCED-98-7 Human Factors

Figure 2: In-the-Air Accidents and Incidents Associated With Pilots’ Performance

                           Accidents a                                                         Incidents

                                                                                                                    Jet exhaust blast 1%
                                                                                                                    Loss of control in flight 1%
                                                                                                                    Engine malfunction 3%
                                                                                                                        Struck ground object
                                                                                                                        (i.e., antenna)
                                                         Aborted                                         6%              Encountered turbulence
                                  15%                    take-off                                                        in flight
                                            4%                                                                12%
 turbulence in      41%                                    Landing gear
 flight                                        9%          not lowered



                          Loss of control in flight

                                                Note: Amounts may not add to 100 percent due to rounding.

                                                    NTSB cited 46 events associated with pilots’ performance in 169 accidents.
                                                    FAA cited 209 events associated with pilots’ performance in 3,901 incidents.

                                                Sources: FAA and NTSB.

                                                In addition to the accidents and incidents discussed above, FAA maintains
                                                data separately for those occasions on which pilots failed to comply with
                                                the air traffic controller’s instructions—such as not staying on the directed
                                                flight path and/or entering a runway without clearance.9 Of the 1,471
                                                unauthorized maneuvers from 1987 through 1995,10 80 percent occurred in
                                                the air, and most of these (73 percent) occurred when pilots did not
                                                maintain their assigned altitude levels. The unauthorized pilot maneuvers
                                                on the ground were most often (69 percent) associated with pilots’ moving
                                                airplanes onto runways without authorization from the air traffic control

                                                 These data are found in FAA’s pilot deviation database.
                                                 These data were available from FAA’s National Aviation Safety Data Analysis Center only for these

                                                Page 7                                                                 GAO/RCED-98-7 Human Factors

                                                 tower. These types of incidents have the potential to cause accidents. For
                                                 example, the December 1990 crash at the Detroit Metropolitan Airport
                                                 occurred when an airplane taxied onto a runway being used for takeoff by
                                                 another airplane and collided with that airplane. Twelve people died. The
                                                 first plane had not gotten permission from the control tower to enter this
                                                 runway, as it should have. Figure 3 shows the most frequently reported
                                                 unauthorized pilot maneuvers in the air and on the ground.

Figure 3: Most Frequently Reported Unauthorized Pilot Maneuvers in the Air and on the Ground

                                  a                                                                   b
      In-the-air maneuvers                                           On-the-ground maneuvers

                                                                                            Take-off on the wrong
                       Other 4%                                                             runway/taxiway 3%
                                                                                                    Landing without clearance
                                        Course clearance deviation                                    Land on the wrong
                                                                                      4%              runway/airport
                                  23%                                                        9%

                                                                                             10%           Take-off without
               73%                                                        69%

  Altitude clearance violation                                       Enter taxi/runway without

                                                  FAA reported 1,110 unauthorized pilot maneuvers in the air.
                                                     FAA reported 258 unauthorized pilot maneuvers on the ground.

                                                 Source: FAA’s pilot deviation database.

Deficiencies in CRM                              In our analysis of accidents, we found deficiencies in the airline pilots’ use
Contributed to Accidents                         of CRM in nearly one-third of all accidents involving pilots’ performance.
                                                 Moreover, we found CRM deficiencies in half of the serious accidents in

                                                 Page 8                                                               GAO/RCED-98-7 Human Factors

which there was at least one fatality. About 46 percent of these CRM
deficiencies involved a lack of coordination among members of the
cockpit crew, as well as the captain’s failure to assign tasks to other crew
members and to effectively supervise the crew. Generally, these CRM
deficiencies illustrated the importance of effective communication.

For example, in the Charlotte, North Carolina, crash in July 1994,
communication among crew members did not occur, according to NTSB’s
accident investigation report. NTSB believes that the captain, who was not
flying the aircraft at the time and could not see the ground because of poor
visibility, became disoriented and commanded the first officer, “down,
push it down,” even though they were encountering windshear, which is a
sudden change in wind direction. The first officer did not question the
order, as he should have, according to NTSB, because the windshear was
creating an unstable situation; the plane could not recover from the
sudden downward shift in direction caused by following the captain’s
order. The plane crashed nose down into the ground, and 37 people died.

Similarly, in a June 1984 accident in Detroit, Michigan, a lack of
communication between the crew and air traffic controllers during a
landing in a severe thunderstorm contributed to the accident, according to
the NTSB report. The crew did not request clarification about the weather
conditions or change its course of action to take these conditions into
account. The winds associated with the storm forced the plane down
precipitously, causing an emergency landing without the landing gear’s
being fully extended. The plane skidded off the runway, causing serious
damage to the aircraft and an emergency evacuation of the passengers.
NTSB reported that the lack of CRM practices was a probable cause of the

The National Aeronautics and Space Administration reported similar
results in its analysis of pilot reports submitted to its voluntary reporting
system.11 Nearly half of the reports cited deficiencies in the pilots’ use of
CRM principles; about 53 percent of the CRM deficiencies concerned
coordination among members, assignment of tasks, and crew supervision.

   An Analysis of part 121 Crew Resource Management Incidents, National Aeronautics and Space
Administration, Quick Response No. 296, Aviation Safety Reporting System (Feb. 6, 1997).

Page 9                                                           GAO/RCED-98-7 Human Factors

                             For AQP training, FAA has specified the process airlines need to follow to
FAA’s Guidance and           develop and implement a curriculum that integrates CRM concepts with
Oversight Do Not             technical flying skills, but FAA’s guidance for CRM training under part 121
Ensure Effectiveness         does not have the same degree of specificity. As a result, inspectors
                             overseeing training under part 121 do not have standards they can use to
of CRM Training              evaluate airlines’ CRM training curriculum and the delivery of that training.
                             Generally, inspectors could not use the guidance provided under AQP to
                             evaluate part 121 training for the CRM curriculum because the curricula
                             developed under the two programs differ significantly. As a result, airlines
                             continue to need specific guidance for CRM under part 121—both those
                             airlines that have opted not to enter AQP as well those that will continue to
                             train at least some of their crews under part 121 until they have fully
                             implemented AQP, which could take up to 8 years.

FAA’s Guidance for CRM       Once an airline elects to participate in AQP, it must follow SFAR 58 (the AQP
Training Is Detailed Under   regulation) for developing a formal curriculum—including assessing the
AQP but Not Under Part       skills pilots need to safely operate the aircraft they fly, developing
                             curriculum objectives for teaching those skills, having measurable criteria
121                          for evaluating whether the pilots have achieved those objectives, and
                             developing materials to teach those objectives. FAA must approve this
                             curriculum. Furthermore, AQP requires all airlines to train their pilots in
                             simulators so that they gain experience with a number of emergency
                             situations. Finally, airlines must submit data to FAA demonstrating that
                             their crews have mastered the skills they need to fly for those airlines.12

                             In developing its AQP curriculum, an airline is required to integrate CRM
                             training into every aspect of its crews’ training. As a result, the pilots
                             trained under AQP are assessed on CRM principles as well as on technical
                             flying skills. For example, when a pilot changes the aircraft’s altitude—a
                             technical flying skill—CRM principles dictate that this pilot inform the
                             other pilot by verbally announcing the new altitude while continually
                             pointing to the altitude indicator until the other pilot also points to the
                             altitude indicator and repeats the new altitude. This procedure is used to
                             ensure that neither pilot will fail to maintain the appropriate altitude.

                             In contrast, FAA’s requirements for CRM training under part 121 do not
                             require airlines to develop a curriculum for CRM training with measurable
                             criteria or to integrate that curriculum with other aspects of part 121
                             training. For the CRM curriculum under part 121, FAA provides suggested

                              Training records for pilots are maintained under part 121. Performance data for crews submitted to
                             FAA under AQP permit the agency to conduct its own analyses of crews’ mastery.

                             Page 10                                                           GAO/RCED-98-7 Human Factors

                           training topics but does not clearly lay out how the airlines are to
                           introduce these topics into their training programs, according to airline
                           officials and FAA inspectors. For example, FAA recommends that airlines
                           train crews in “workload management and situational awareness.” For this
                           training, FAA suggests such topics as “preparation/planning/vigilance” and
                           “workload distribution/distraction avoidance.” However, for those airlines
                           that choose to integrate these topics with technical flying skills, FAA does
                           not explain how the airlines are to do so.13

                           The lack of specificity in FAA’s guidance for the development of a CRM
                           curriculum under part 121 contrasts with the detailed guidance FAA
                           provides for the development of a curriculum on technical flying skills.
                           For example, FAA’s guidance on how pilots are to respond to windshear
                           under part 121 directs them in a number of technical flying skills, such as
                           how to handle the rudder, but it is silent on how to employ CRM principles
                           in this situation. In contrast, under AQP, FAA’s guidance instructs the
                           airlines to specify not only the technical skills but also the CRM principles
                           that must be applied in a windshear situation.

                           Because FAA’s guidance on CRM training under part 121 is less specific,
                           airlines vary in how they deliver their CRM training. While all the airlines
                           provide classroom training in CRM principles under part 121 training, they
                           may not integrate this training with technical flying skills. For example,
                           airlines may (1) train pilots in technical flying skills in flight simulators
                           without integrating CRM principles or (2) integrate CRM principles with
                           technical flying skills in flight simulators. Generally, we found that CRM
                           training had been integrated with technical flight training to a higher
                           degree at those airlines that were in later phases of AQP implementation.

FAA’s Oversight of CRM     In developing AQP, FAA incorporated procedures for evaluating CRM training
Training Is Adequate for   and developed a process for ensuring that FAA inspectors would have the
AQP but Not for Part 121   criteria they need to conduct the evaluations for pilots’ training on
                           different types of aircraft. Specifically, AQP provides a systematic way of
                           identifying the tasks and subtasks involved in a particular phase of flight.
                           Therefore, an inspector observing the training program can determine
                           whether CRM principles are being invoked in a given flight situation. For
                           example, when a crew is preparing for landing, AQP specifies that the first
                           officer, if unsure of the planned course of action in the event of a missed
                           approach, is to ask the captain to clarify the plan so that both have a full
                           understanding of the actions they will take. Similarly, if a flight has to be

                             See FAA’s Advisory Circular 120-51B, Crew Resource Management Training.

                           Page 11                                                        GAO/RCED-98-7 Human Factors

diverted from one airport to another, the captain is to direct the first
officer to (1) get out the maps for the alternate airport, (2) notify the flight
attendants, and (3) make the announcement to the passengers. This
delegation of tasks allows the captain time to handle radio contact with
the airline’s dispatchers and air traffic controllers, obtain weather updates
at the alternate airport, and fly the plane.

In the early stages of AQP implementation, the AQP Branch is evaluating
airlines’ training. FAA will transfer this responsibility to inspectors in the
field as airlines fully implement AQP. Field inspectors will be trained in
evaluating the CRM training as an integral part of their evaluation of AQP
training. The inspectors at those airlines that had progressed beyond the
initial phases of AQP noted that they had received AQP training at the
airlines for which they were responsible. Moreover, all of the inspectors
we spoke with maintained that while certain facets of AQP were fixed,
some parts were still evolving. As a result of the program’s flexibility and
evolution, the inspectors pointed out that it was not possible to structure a
training program for them that could cover every aspect of AQP at every
airline. Despite this fluidity, these inspectors said that the AQP Branch
Office made sure that the program’s standards were maintained across

While the evaluation of the delivery of CRM training is incorporated into the
oversight process for AQP training, it is not under traditional part 121
training. Moreover, FAA has not provided its inspectors with any specific
guidance or training for evaluating airlines’ CRM training under part 121.
Although FAA inspectors may obtain some CRM training from a 3-hour
computerized interactive course, this lack of guidance for evaluating CRM
training under part 121 is troublesome to the inspectors we spoke with
because of what they view as an inherent conflict between performance
expectations for individuals under part 121 and crew performance
expectations articulated in CRM principles.14 Under part 121, pilots are to
master technical flying skills and perform these skills without reliance on
any other crew member. In contrast, CRM principles and training teach
pilots how to use to maximum effect the abilities and experience of other
crew members, as well as their own technical flying skills.

Without formal FAA instructions, inspectors have developed their own
approaches to this evaluation. For example, one inspector said that he
based his approval on his belief that the airline for which he was

 In a previous review of FAA’s training for its inspector workforce—Aviation Safety: Targeting and
Training of FAA’s Safety Inspector Workforce (GAO/T-RCED-96-26, Apr. 30, 1996)—we found that
some inspectors were unaware that needed training was available through computer-based courses.

Page 12                                                           GAO/RCED-98-7 Human Factors

responsible “had a good safety record” and “would probably establish a
good program.” Another inspector said that in approving any training
program, he sought guidance first from any applicable federal aviation
regulation; the Inspector’s Handbook, applicable advisory circulars; and,
finally, any other FAA publication, such as the Introduction to CRM Training.
However, this inspector added that these sources did not provide the
criteria he needed to evaluate CRM training. As a result, he looked for
behaviors such as crew members’ “working together” to resolve problems,
“catching errors,” or “dealing with the consequences resulting from
uncaught errors.”

These ad hoc approaches to evaluating the delivery of CRM training are not
satisfactory to FAA officials at headquarters or to officials for at least one
airline. FAA officials told us that the agency needed additional CRM training
for its inspectors conducting reviews under part 121. In addition, officials
at one airline told us that the lack of specific guidance and training for FAA
inspectors responsible for evaluating CRM training under part 121 has
hampered FAA’s ability to review CRM programs. Furthermore, the problems
FAA inspectors face in evaluating CRM training under part 121 will continue
indefinitely in the absence of clearer guidance from FAA for those airlines
that have decided not to enter AQP and for those airlines in the program
that have not fully implemented it.

Because AQP is implemented by the type of aircraft the crew flies, even the
airlines that have been accepted for AQP will continue to provide some CRM
training under part 121. For the eight airlines implementing AQP, we
estimate that only about one-third of their crews have begun to receive AQP
training. Therefore, most crews are still receiving traditional training
under part 121, and some will continue do so for up to 8 years. As of
September 1997, the airlines’ estimated dates for completing the transition
to AQP training ranged between 2000 and 2005. (See table 1.)

Page 13                                            GAO/RCED-98-7 Human Factors

Table 1: Estimate of Completed AQP
Implementation by Airline                                                                           Pilots being
                                                                                                  trained under
                                                                                                      AQP as of Estimated year of
                                     Airline                                 Year begun         September 1997        completion
                                     Alaska                                           1994                        0                   2002
                                     American                                         1994                        0                   2005
                                     Continental                                      1997                        0                   2002
                                     Delta                                            1992                       60%                  2001
                                     Northwest                                        1994                       23%                  2001
                                     Trans World                                      1995                        0                   2000
                                     United                                           1991                       50%                  2001
                                     US Airwaysa                                      1994                        0                   2000
                                      Estimate is for one aircraft type. US Airways’ aircraft fleet composition is under review, and no
                                     determination has been made to place any other aircraft types into AQP until the aircraft fleet plan
                                     is resolved.

                                     For the flying public, safety is the paramount issue, and FAA and the
Conclusions                          airlines have worked to provide rigorous training programs for pilots.
                                     Crew resource management, which focuses on making the best use of all
                                     available experience and skills in the cockpit, is increasingly seen as an
                                     important component of safe flights. FAA recognized the importance of
                                     crew resource management by requiring all airlines to include training in
                                     these principles and by incorporating crew resource management into its
                                     Advanced Qualification Program.

                                     Pilots’ performance is not the only factor in airline accidents, but it is an
                                     important one. We identified pilots’ performance as the cause of about
                                     one-third of all the accidents and nearly one-fifth of the incidents for the
                                     10 major airlines from 1983 through 1995.

                                     Training for safer performance by pilots that teaches crew resource
                                     management can occur under either the Advanced Qualification Program
                                     or part 121. However, while FAA’s guidance for the implementation of the
                                     Advanced Qualification Program specifies a process for curriculum
                                     development that integrates this training with training in technical flying
                                     skills, FAA’s guidance for curriculum development under part 121 is
                                     ambiguous and does not provide standards that inspectors can use to
                                     evaluate and approve airlines’ training in crew resource management. As a
                                     result, FAA cannot be assured that airlines are developing a curriculum for
                                     teaching crew resource management that will effectively teach pilots how

                                     Page 14                                                            GAO/RCED-98-7 Human Factors

                     to best use all the skills and experience available to them in the cockpit.
                     Furthermore, without specificity in the development of training for crew
                     resource management under part 121 and without any guidance on how to
                     evaluate this training under part 121, FAA inspectors are relying on their
                     own experience in observing pilots or even on the belief that the airline
                     “would probably establish a good program.” These problems are especially
                     troublesome because pilots who have not completed FAA-approved crew
                     resource management training by March 1998 may not fly for airlines.

                     To help ensure that airlines appropriately train pilots in CRM principles
Recommendations      under part 121 and that FAA inspectors are able to uniformly evaluate this
                     CRM training, we recommend that the Secretary of Transportation direct
                     the Administrator of FAA to develop a process that airlines must follow for
                     creating a CRM curriculum, with measurable criteria, under part 121 as it
                     has for the Advanced Qualification Program.

                     We provided a draft of this report to FAA for review and comment. We met
Agency Comments      with the Deputy Associate Administrator for Regulation and Certification,
and Our Evaluation   the Deputy Director of Flight Standards Services, the Managers for the Air
                     Carrier Training Branch and the Advanced Qualification Program, and
                     other officials. FAA commended our review of CRM training at the nation’s
                     airlines. FAA accepted the report’s recommendation in part. FAA agreed that
                     it should ensure that pilots are appropriately trained and noted that CRM
                     training can provide desirable consequences in aviation safety. It further
                     agreed that uniform evaluation of CRM training using measurable criteria is
                     a commendable objective. However, FAA stated that science has not yet
                     developed valid, reliable criteria for measuring CRM performance. FAA also
                     agreed that more can be done to develop a process that airlines and
                     inspectors can follow to create a CRM curriculum. FAA indicated that better
                     guidance would be provided in a number of ways, such as updating
                     Advisory Circular 120-51, Crew Resource Management Training, and
                     supplemental guidance for inspectors included in the inspectors’
                     handbook and holding regional meetings with CRM specialists from Flight
                     Standards Services and other organizations.

                     We concur with FAA that CRM training for pilots could improve aviation
                     safety. However, we believe that before the contribution of CRM training to
                     aviation safety can be measured, it is necessary to determine the extent to
                     which the delivery of CRM training for pilots has occurred. We further
                     concur with FAA that more should be done to develop processes for

                     Page 15                                           GAO/RCED-98-7 Human Factors

              airlines and inspectors to follow in creating a CRM curriculum. We believe
              that until FAA establishes a process for CRM curriculum development that
              includes an assessment of the extent to which pilots have mastered that
              curriculum, it will not be possible to measure CRM’s performance in
              contributing to aviation safety.

              To determine the extent to which inadequate performance by pilots was a
Scope and     problem for the 10 major U.S. airlines, we examined the types and
Methodology   frequency of safety-threatening events—incidents and accidents—from
              1983 through 1995.

              To determine the adequacy of FAA’ s guidance for and oversight of pilots’
              training, we reviewed FAA’s role in the airlines’ implementation of CRM. We
              focused primarily on CRM training because FAA has described the failure to
              apply CRM principles as a more important contributing factor in accidents
              than technical flying skills. We also compared FAA’s rules and regulations
              and other guidance for CRM training with that provided for other training
              programs, as well as interviewed FAA and airline officials. A detailed
              discussion of our methodology is presented in appendix I. Related GAO
              products are listed at the end of this report.

              Our work was performed from October 1996 through October 1997 in
              accordance with generally accepted government auditing standards.

              As arranged with your offices, unless you publicly announce its contents
              earlier, we plan no further distribution of this report until 30 days after the
              date of this letter. At that time, we will provide copies of the report to the
              Secretary of Transportation, appropriate congressional committees, and
              other interested parties. We will also make copies available to others upon

              Page 16                                             GAO/RCED-98-7 Human Factors

Please contact me at (202) 512-3650 if you or your staff have any questions
about this report. Major contributors to this report are listed in appendix

Gerald L. Dillingham
Associate Director, Transportation

Page 17                                          GAO/RCED-98-7 Human Factors

Letter                                                                                           1

Appendix I                                                                                      20

Appendix II                                                                                     23

Major Contributors to
This Report
Related GAO Products                                                                            24

Tables                  Table 1: Estimate of Completed AQP Implementation by Airline            14
                        Table I.1: Classification Framework for Categorizing CRM Issues         21

Figures                 Figure 1: On-the-Ground Accidents and Incidents Associated               6
                          With Pilots’ Performance
                        Figure 2: In-the-Air Accidents and Incidents Associated With             7
                          Pilots’ Performance
                        Figure 3: Most Frequently Reported Unauthorized Pilot                    8
                          Maneuvers in the Air and on the Ground


                        AQP        Advanced Qualification Program
                        ASRS       Aviation Safety Reporting System
                        CRM        crew resource management
                        FAA        Federal Aviation Administration
                        GAO        General Accounting Office
                        NASA       National Aeronautics and Space Administration
                        NTSB       National Transportation Safety Board

                        Page 18                                         GAO/RCED-98-7 Human Factors
Page 19   GAO/RCED-98-7 Human Factors
Appendix I


              To identify the types and frequencies of accidents and
              incidents—safety-threatening events—related to pilot performance, we
              reviewed accident and incident data, including pilot deviations, contained
              in the National Transportation Board’s (NTSB) and the Federal Aviation
              Administration’s (FAA) electronic databases. We obtained these data from
              FAA’s National Aviation Safety Data Analysis Center. We limited our review
              to the reported events in accident and incident data sources involving the
              10 major U.S. passenger airlines from 1983 through 1995. We did not
              independently verify these data.

              To facilitate the comparison of accidents with incidents in our analysis of
              the types and frequencies of safety-threatening events, we made two
              adjustments to the data. First, because of differences in the way
              information is recorded in these databases, we matched the similar
              categories contained in both databases and used these categories in our
              analysis. For example, both NTSB’s and FAA’s databases contain the
              category “on ground collision with object,” which means an airplane
              struck an object, such as a vehicle or structure, while moving on the
              ground. Second, because the occurrences of events in accidents closely
              conform to those in incidents, we used the events that occurred in each of
              the 169 accidents as our unit of analysis. In our analysis of crew resource
              management (CRM) deficiencies, we used the accident as the unit of
              analysis because NTSB’s findings of CRM deficiencies were by accident and
              not by the individual events that occurred within accidents.

              To characterize the prevalence of pilot performance as a factor in
              safety-threatening events over time and between airlines, we examined
              FAA’s incident and pilot deviation databases. We used these two databases
              because they are the only such sources with adequate numbers of
              observations to make such comparisons.

              To determine the extent to which the inadequate use of CRM by pilots
              contributed to accidents and incidents, we performed a content analysis of
              the textual information found in the factual reports, briefs, and final
              reports of the 169 accidents investigated by NTSB from 1983 through 1995.
              We then classified CRM deficiencies according to the classification
              framework presented at a National Aeronautics and Space Administration
              (NASA)/Ames workshop in 1980.15 This framework groups CRM issues into
              five broad clusters:

               Murphy, M.R. (1980). “Analysis of Eighty-four Commercial Aviation Incidents: Implications for a
              Resource Management Approach to Crew Training.” 1980 Proceedings Annual Reliability and
              Maintainability Symposium. Ames Research Center, NASA, Moffet Field, California.

              Page 20                                                           GAO/RCED-98-7 Human Factors
                                            Appendix I

                                        •   (1) Resource management—the application of specialized cognitive skills
                                            to effectively and efficiently utilize available resources, such as the ability
                                            to plan, organize, and communicate.
                                        •   (2) Organization processes—crew members’ actions and behaviors in the
                                            context of their assigned duties and expected responsibilities.
                                        •   (3) Personal factors—the knowledge, skills, abilities, and limitations that
                                            individual crew members bring with them to the cockpit.
                                        •   (4) Material resources internal to the aircraft—the cockpit crew’s
                                            appropriate, effective, and efficient use of instructional items, such as
                                            checklists, and navigational charts and equipment, such as on-board
                                            weather radar, navigational controls, and engine fire extinguisher.
                                        •   (5) Resources external to the aircraft—those people (air traffic
                                            controllers), entities (airports), and circumstances (emerging poor
                                            weather) that may affect pilots’ plans, decisions, and actions.

                                            Table I.1 shows the classification framework used to categorize CRM

Table I.1: Classification Framework for Categorizing CRM Issues
                     Resource                Organization                            internal to the     Resources external
                     management              processes            Personal factors   aircraft            to the aircraft
CRM issues           •Social                 •Role                •Knowledge         •Textual            •Human
                     •Communication          •Monitoring          •Proficiency       •Equipment          •Facility
                     •Leadership             •Workload            •Experience                            •Environment
                     •Management                                  •Motivation
                     •Planning                                    •Stress
                     •Problem-solving                             reaction
                     •Decision-making                             •Fatigue

                                            To verify the results of our content analysis, we requested a similar
                                            analysis by NASA’s Aviation Safety Reporting System (ASRS) staff of
                                            voluntarily submitted pilot reports contained in the ASRS database.
                                            According to the aviation experts we consulted, ASRS incident reports
                                            provide the best source of information on deficiencies in CRM.
                                            Furthermore, because ASRS staff are most familiar with the data and have
                                            expertise in analyzing this free-form data, we concluded that it was more
                                            appropriate for them to perform this analysis.

                                            To evaluate the adequacy of FAA’s oversight of airline pilot training, we
                                            obtained FAA’s training policies, requirements, guidance, and handbooks

                                            Page 21                                               GAO/RCED-98-7 Human Factors
Appendix I

relevant to CRM training. We discussed training programs, including CRM,
and training procedures with appropriate FAA officials, including officials
in the Office of System Safety, the Office of Regulation and Certification’s
Flight Standards Services, the Advanced Qualification Program Branch,
the Office of Accident Investigation, and the Human Factors Division. In
addition, we discussed airline training evaluation and approval processes
and obtained training documents from FAA inspectors responsible for
monitoring airline training. Finally, we contacted safety directors and
trainers at the major airlines and obtained documents on their policies,
procedures, research, and training curricula.

We requested comments from recognized experts in the field of human
factors in academia and the aviation industry, pilots, and government
officials from FAA, NTSB, and NASA. We incorporated their comments where
appropriate and made adjustments to our methodology as warranted.

Page 22                                           GAO/RCED-98-7 Human Factors
Appendix II

Major Contributors to This Report

                        John H. Anderson, Jr.
Resources,              Emilie G. Heller
Community, and          Luann M. Moy
Economic                Richard R. Scott
                        E. Jerry Seigler
Development             Carol Herrnstadt Shulman
Division, Washington,   Robert E. White
                        Mindi G. Weisenbloom
Office of General
                        Christopher A. Keisling
Atlanta Regional

                        Page 23                    GAO/RCED-98-7 Human Factors
Related GAO Products

              Aviation Safety: New Airlines Illustrate Long-Standing Problems in FAA’s
              Inspection Program (GAO/RCED-97-2, Oct. 17, 1996).

              Human Factors: Status of Efforts to Integrate Research on Human Factors
              Into FAA’s Activities (GAO/RCED-96-151, June 27, 1996).

              Military Aircraft Safety: Significant Improvements Since 1975
              (GAO/NSIAD-96-69BR, Feb. 1, 1996).

              Aviation Safety: Data Problems Threaten FAA Strides on Safety Analysis
              System (GAO/AIMD-95-27, Feb. 8, 1995).

              Aviation Safety: Unresolved Issues Involving U.S.-Registered Aircraft
              (GAO/RCED-93-135, June 18, 1993).

              Aviation Safety: Changes Needed in FAA’s Service Difficulty Reporting
              Program (GAO/RCED-91-24, Mar. 21, 1991).

(341513)      Page 24                                          GAO/RCED-98-7 Human Factors
Ordering Information

The first copy of each GAO report and testimony is free.
Additional copies are $2 each. Orders should be sent to the
following address, accompanied by a check or money order
made out to the Superintendent of Documents, when
necessary. VISA and MasterCard credit cards are accepted, also.
Orders for 100 or more copies to be mailed to a single address
are discounted 25 percent.

Orders by mail:

U.S. General Accounting Office
P.O. Box 37050
Washington, DC 20013

or visit:

Room 1100
700 4th St. NW (corner of 4th and G Sts. NW)
U.S. General Accounting Office
Washington, DC

Orders may also be placed by calling (202) 512-6000
or by using fax number (202) 512-6061, or TDD (202) 512-2537.

Each day, GAO issues a list of newly available reports and
testimony. To receive facsimile copies of the daily list or any
list from the past 30 days, please call (202) 512-6000 using a
touchtone phone. A recorded menu will provide information on
how to obtain these lists.

For information on how to access GAO reports on the INTERNET,
send an e-mail message with "info" in the body to:


or visit GAO’s World Wide Web Home Page at:


United States                       Bulk Rate
General Accounting Office      Postage & Fees Paid
Washington, D.C. 20548-0001           GAO
                                 Permit No. G100
Official Business
Penalty for Private Use $300

Address Correction Requested