oversight

Aviation Safety: Research Supports Limited Use of Personal Computer Aviation Training Devices for Pilots

Published by the Government Accountability Office on 1999-07-12.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                  United States General Accounting Office

GAO               Report to Congressional Requesters




July 1999
                  AVIATION SAFETY
                  Research Supports
                  Limited Use of
                  Personal Computer
                  Aviation Training
                  Devices for Pilots




GAO/RCED-99-143
      United States
GAO   General Accounting Office
      Washington, D.C. 20548

      Resources, Community, and
      Economic Development Division

      B-280735

      July 12, 1999

      The Honorable John J. Duncan
      Chairman, Subcommittee on Aviation
      Committee on Transportation and Infrastructure
      House of Representatives

      The Honorable Thomas W. Ewing
      House of Representatives

      In 1998, general aviation had 1,907 accidents, with 621 fatalities. The
      National Transportation Safety Board estimates that 87 percent of all fatal
      general aviation accidents are caused by pilot error, especially when pilots
      who do not have appropriate instrument training fly when visibility is
      poor, such as during bad weather. To reduce the occurrence of general
      aviation accidents, the Federal Aviation Administration (FAA) has been
      exploring a number of means to enhance the training of general aviation
      pilots.

      One possible enhancement is the use of new technologies for the training
      that occurs on the ground. For over 40 years, general aviation student
      pilots have used flight training devices to help them learn how to fly using
      an aircraft’s instruments alone. These flight-training devices resemble an
      aircraft’s cockpit and are often constructed with actual airplane
      instruments; they can be used by student pilots to substitute for up to 20 of
      the 40 hours of airplane training required by FAA to obtain an instrument
      rating. The instrument rating permits a pilot to fly when visibility is poor.
      In May 1997, FAA also approved the use of special personal computers,
      controls, and software called personal computer-based aviation training
      devices (PCATD), which can be used for up to 10 hours of instrument
      training.1 FAA’s decision to allow the use of PCATDs has sparked debate.
      Some assert that pilots trained with these devices will be less skilled,
      thereby compromising aviation safety. Others argue that pilots trained
      with the devices are actually better trained at lower cost.

      As a result of this controversy, you asked us to (1) describe the process
      and information FAA used in deciding to approve the use of personal

      1
       “Personal Computer-Based Aviation Training Device,” or PCATD, is FAA’s term for computer-based
      devices that have been approved for use, or qualified for credit hour training, by FAA officials. FAA
      requires PCATDs to be qualified, approved, and used in connection with an integrated ground and
      flight instrument training curriculum. Training using PCATDs should include the procedural tasks
      listed in FAA’s Advisory Circular 61-126. In this report, we refer to these devices as computer-based
      devices before they were approved for use in training and PCATDs once the devices were approved for
      use by FAA. PCATDs describes one of the four FAA-approved devices.



      Page 1                  GAO/RCED-99-143 Personal Computer-Based Aviation Training Device
                   B-280735




                   computer devices for 10 hours of instrument training and (2) discuss what
                   is known about the training effectiveness of these devices and their
                   long-term impact on a pilot’s ability to fly safely. To respond to these
                   objectives, we interviewed federal officials and others familiar with FAA’s
                   decision and reviewed FAA’s supporting documentation; conducted a
                   comprehensive literature search on the use of these and similar
                   computer-based devices, on instrument training, and on general aviation
                   safety; reviewed about 700 studies and articles; analyzed and summarized
                   the most relevant data-based literature; interviewed government,
                   academic, and private sector flight instruction experts on the use of the
                   devices; and identified other issues related to the devices’ potential
                   impacts on aviation safety.


                   FAA’s decision to allow the use of computer-based devices for instrument
Results in Brief   flight training took over 6 years to be made final and was based on two
                   major research studies, FAA’s professional judgment, and input from
                   aviation industry representatives. Because the two studies did not address
                   the appropriate number of hours of training on these devices, FAA’s
                   decision to allow 10 hours was based on its professional judgment and
                   industry input. FAA had earlier relied on its professional judgment, rather
                   than empirical studies, to support its approval of up to 20 credit hours for
                   the devices now known as flight training devices. A University of Illinois
                   research study is planned to assess the appropriate number of credit hours
                   for the use of these devices in an instrument training course.

                   The two major research studies generally support the use of
                   computer-based devices for training. Despite some methodological
                   limitations, these are the most complete controlled studies to date on the
                   training effectiveness of computer-based devices. One study shows that
                   the use of computer-based devices may modestly reduce the training time
                   spent in an airplane, while the other shows that the training effects of
                   computer-based devices and previously approved flight training devices
                   may not differ greatly. Other studies we reviewed also generally supported
                   the use of computer-based devices in training. Similarly, the majority of
                   the experts we interviewed saw some training value from the use of the
                   devices and did not believe that they were likely to reduce aviation safety.
                   Although most experts did not speculate on the appropriate number of
                   credit hours that should be granted for the devices, several disagreed with
                   FAA’s decision to allow any credit hours. We found no empirical evidence,
                   however, on the long-term safety impact of the devices, their potential
                   safety benefits, or the long-term safety impact or benefits of the currently



                   Page 2            GAO/RCED-99-143 Personal Computer-Based Aviation Training Device
             B-280735




             approved flight training devices. Moreover, FAA does not currently collect
             the data needed to conduct future research on a possible link between the
             use of the devices and pilots’ long-term safety records.

             This report recommends that FAA gather additional information on the
             long-term safety issues associated with computer-based devices and
             previously approved flight training devices. FAA will soon revise and
             computerize its pilot application form. This form could collect additional
             information at minimal cost on the use of pilot training devices and would
             facilitate future research.


             To establish training requirements for pilots, FAA issues regulations for
Background   pilots, pilot schools, instructors, and pilot training equipment
             manufacturers, and provides guidance through advisory circulars,
             handbooks, and other types of informational material.2 Within FAA, several
             organizations regulate pilot training. For example, the Flight Standards
             Office is responsible for certifying and overseeing pilot schools, pilots, and
             pilot training equipment. Likewise, FAA’s Civil Aeromedical Institute (CAMI)
             studies human performance factors in aviation, among other issues.
             Finally, FAA’s National Simulator Program studies and evaluates the more
             sophisticated simulator equipment used in training pilots.

             Beginning students first learn to fly under FAA’s visual flight rules, which
             generally require a minimum visibility of 3 miles, although most new pilots
             fly with much better visibility. Pilots then must obtain an instrument rating
             to fly in conditions of reduced visibility, such as in clouds or in poor
             weather. Flying when visibility is poor is more difficult because a pilot
             loses reference to the outside horizon and must rely completely on
             instruments to know the plane’s spatial orientation. Instrument training is
             important to aviation safety because the vast majority of accidents involve
             poor weather and poor visibility. When a pilot without instrument training
             mistakenly flies into conditions of poor visibility, disorientation and loss of
             control can occur quickly; many fatal weather-related accidents occur
             when a pilot who is flying according to visual flight rules accidentally
             encounters poor visibility conditions that are beyond his or her abilities.
             According to experts, good instrument skills are critical to flying safely
             through bad weather.




             2
             The regulation of pilots, pilot schools, and pilot training equipment by FAA is governed by title 14,
             C.F.R. parts 61, 91, 121, 135, and 141.



             Page 3                   GAO/RCED-99-143 Personal Computer-Based Aviation Training Device
B-280735




An instrument rating is considered difficult to obtain and normally
requires additional training, an FAA written exam, and a practical
test—called a “check ride”—with an FAA-designated examiner in an
aircraft. Before taking these exams, pilots must take an additional 35 to 40
hours of training, which may include up to 20 hours in a flight simulator or
flight training device, with the remaining time in an aircraft.3 However,
pilots typically take about 70 hours of training in an aircraft or training
device before they are proficient enough to pass the tests needed to obtain
their instrument rating.

In May 1997, FAA issued an advisory circular approving the use of PCATDs
for training credit toward the instrument rating.4 These aviation training
devices are personal computers with specialized software and add-on
equipment, including separate simulated radio and navigational controls to
the right of the computer monitor, a throttle, a steering yoke under the
computer screen, and rudder pedals on the floor. A fully equipped
FAA-approved PCATD costs from $5,000 to $10,000. In comparison, a
previously approved flight training device costs from $40,000 to $90,000.5
When training with either a PCATD or a flight training device, the instructor
usually sits with a student, sometimes at a separate personal computer
console where the instructor can plan, monitor, and control a student’s
flight. Figure 1 shows an FAA-approved PCATD.




3
 There are two types of pilot training schools. One is an FAA-approved school (Part 141) and others are
nonapproved schools that still must meet FAA training requirements (Part 61). An FAA-approved school
may be authorized to give its graduates written exams and check rides. In addition, an FAA-approved
school can qualify pilots with fewer hours than can a nonapproved school. Many of the nonapproved
pilot schools find it impractical to qualify for FAA certificates, but 85 percent of all instrument ratings
are granted for the training conducted at these schools.
4
 FAA Advisory Circular 61-126.
5
 In contrast, a full-motion simulator used by airlines to train commercial pilots can cost up to
$20 million.



Page 4                   GAO/RCED-99-143 Personal Computer-Based Aviation Training Device
                                    B-280735




Figure 1: Personal Computer-Based
Aviation Training Device




                                    Source: Aviation Supplies and Academics, Inc.




                                    Page 5                 GAO/RCED-99-143 Personal Computer-Based Aviation Training Device
                              B-280735




                              FAA’s advisory circular spells out the conditions that an aviation training
                              device must meet in order to be qualified by FAA as a PCATD for use in
                              satisfying the instrument rating requirements. The advisory circular’s
                              guidelines are not mandatory and contain a voluntary reporting
                              mechanism requesting information on students’ PCATD use. FAA could have
                              used its rule-making authority to approve PCATD use. It did not do so for
                              several reasons, according to Flight Standards officials. First, rules are
                              mandatory and advisory circulars are not, making any significant changes
                              to mandatory rules more difficult and time-consuming than making any
                              significant changes to advisory circulars. Second, in the past, the
                              qualification of ground flight training devices in general aviation was
                              handled by issuing advisory circulars. Finally, FAA officials wanted to
                              facilitate the rapid introduction of the new PCATD technology without going
                              through a lengthy rule-making process. Once FAA has qualified a
                              PCATD—including its hardware, software, and accompanying
                              equipment—the device can be used by a flight school or an individual
                              flight instructor in an integrated ground and flight training curriculum for
                              up to 10 hours of flight instruction toward an instrument rating.


FAA’s Decision to Allow       FAA’s decision to allow the use of computer-based devices and to grant
the Use of Computer-Based     credit hours evolved over many years. Since the 1980s, FAA had been
Devices Was Based on Two      evaluating various designs of PCATDs but did not consider these early
                              devices to be sophisticated enough for general use in instrument flight
Studies, FAA’s Professional   courses. As computer equipment advanced and training software was
Judgment, and Industry        refined, FAA in 1991 began to seriously consider proposals from
Input                         manufacturers and aviation schools to allow the use of computer-based
                              devices for instrument flight training. Initially, FAA officials considered
                              allowing the use of the devices for general instructional purposes only and
                              not for students to earn flight hours toward an instrument rating.

                              In 1992, Flight Standards officials proposed developing a research project
                              to obtain information on the use of the devices because they were
                              concerned that they needed empirical data to support the introduction of
                              this new training technology. According to Flight Standards officials, FAA
                              has never sponsored a study that evaluated the performance of previously
                              approved flight training devices, which remain in wide use. FAA relied on
                              its professional judgment rather than empirical studies to support the
                              approval of up to 20 credit hours for flight training devices.

                              Later in 1992, FAA provided a grant to Embry-Riddle Aeronautical
                              University to study the use of computer-based devices. A principle



                              Page 6            GAO/RCED-99-143 Personal Computer-Based Aviation Training Device
B-280735




objective of the study was to measure the transfer of instrument training
knowledge to an aircraft for three groups of students trained using two
different computer-based training devices and one approved flight training
device. According to Flight Standards officials, the Embry-Riddle study
was designed to provide them with information on whether the devices
were useful and effective. In June 1993, while the Embry-Riddle research
was ongoing, CAMI’s Human Factors Research Laboratory submitted a
research proposal to expand and participate in the study. This detailed
approach was not approved, according to a Flight Standards official,
because FAA at the time did not have the funds to support it. For the next
year, CAMI officials continued to help explore possible research
methodologies for qualifying the devices in a structured training
curriculum that would provide some insight into determining if credit
hours should be granted and what tasks the devices are best suited for in
instrument training. However, at that time, FAA was not proposing to grant
flight training credit hours for training on computer-based devices.

When the Embry-Riddle study was completed in 1994, some FAA officials
outside of Flight Standards believed that the study had significant
limitations, was not properly designed, and did not fully answer important
questions on whether computer-based devices should be approved. CAMI
officials stated that the failure to include a control group that received
training only in the aircraft seriously limited the usefulness of the study. In
addition, officials in FAA’s National Simulator Program stated that no
meaningful conclusion could be drawn from the research.

In June 1994, FAA sponsored a meeting of manufacturers and others in the
aviation training industry to obtain their views on FAA’s current and future
direction in approving the use of computer-based devices. In addition,
starting in 1994, FAA sponsored further research through a series of grants
to the University of Illinois to study the training effectiveness of the
devices and to answer many of the questions and concerns about their use
that remained following the Embry-Riddle study.

While FAA was evaluating the qualifications of computer-based devices for
instrument training without credit hours, it received requests to allow
some of the training hours done on the equipment to be credited toward
the minimum hours of flight time in instrument training. Some
manufacturers, for example, wanted FAA to consider granting credit hours
in order to make their equipment more credible to their customers.
Furthermore, in May 1995, the Aircraft Owners and Pilots Association’s Air
Safety Foundation formally petitioned FAA to allow training on



Page 7             GAO/RCED-99-143 Personal Computer-Based Aviation Training Device
B-280735




computer-based devices to count for training time toward an instrument
rating and proposed a syllabus illustrating the integration of the devices
into an instrument flight course.

Although neither the Embry-Riddle nor the University of Illinois study
addressed any appropriate number of flight training credit hours, FAA
officials used their experience and professional judgment to support their
decision and saw it as a reasonable compromise between the various
proposals. Furthermore, FAA officials decided to add a voluntary reporting
mechanism to the draft advisory circular designed to gather information to
validate the approval and use of the devices with feedback from users.
Finally, FAA officials told us that the required check ride by an
FAA-designated examiner after a student’s training would ensure that all
instrument students were qualified to receive the instrument rating, no
matter what equipment they used during training. Several flight training
experts we interviewed, however, told us that the check ride is an
imperfect measure of a pilot’s ability, should not be seen as assurance of
proper training, and is instead a snapshot of a pilot on a single day. These
experts believe assurances on the quality of training devices, a quality
curriculum, capable instructors, consistent testing by FAA-designated flight
examiners, and long-term research on pilot safety and training are needed.

In August 1995, FAA sponsored an additional meeting of aviation industry
officials and presented a draft advisory circular that proposed, for the first
time, granting 10 flight training credit hours. The draft advisory circular
and all significant changes to the draft were submitted to over 60 groups,
organizations, and other interested parties. For the next 12 months, FAA
officials received numerous comments on the use of computer-based
devices. During this time, several aviation industry officials suggested that
FAA delay or withhold approval of a final advisory circular pending more
research and the completion of the University of Illinois study. In
September 1996, FAA decided to delay approving the advisory circular until
after the University of Illinois study was completed.

In November 1996, the University of Illinois issued its final report and in
March 1997 briefed FAA officials on the results. The study concluded, in
part, that computer-based devices can provide a worthwhile training
benefit and can save a small portion of the aircraft time that would
otherwise be needed. In January 1997, the National Air Transportation
Association, after extensive discussion, recommended that FAA approve a
draft advisory circular, including flight training credit hours for
computer-based devices, as well as other recommendations.



Page 8            GAO/RCED-99-143 Personal Computer-Based Aviation Training Device
B-280735




In May 1997, FAA issued Advisory Circular 61-126, which created a separate
category of training device—PCATD—rather than integrating PCATDs into the
“ground training device” category already approved for credit hours in
instrument training. These existing ground training devices were renamed
flight training devices (FTD) to distinguish them from PCATDs. Older models
of the new FTD category were grandfathered in and automatically approved
by FAA for instrument credit hours. According to Flight Standards officials,
the decision to approve 10 hours of flight credit was based on (1) the
general utility of computer-based devices in instrument training, as shown
by the Embry-Riddle and University of Illinois studies; (2) a wide variety of
proposals, ranging from zero hours of credit to 20 hours; and (3) the input
from manufacturers that they needed credit hours to sell their equipment.
Two additional research studies are planned: One will try to assess the
appropriate number of credit hours for PCATD use in an instrument course;
the second will investigate the use of PCATDs to maintain instrument skills.6


Over the next several months, FAA officials qualified four PCATD devices,
which included three software programs and several other manufacturers
of controls, while finding an equal number unacceptable. As of June 1999,
no information had been sent to FAA under the advisory circular’s
voluntary reporting mechanism on the use of PCATDs. An official of the
National Association of Flight Instructors estimated that very few of the
200 Part 141 flight schools now use PCATDs for instrument instruction.




6
 This University of Illinois research study is likely to be funded by the National Aeronautics and Space
Administration’s Ames Research Laboratory, which investigates human factors issues and funds
research investigating general and commercial aviation safety. The study proposes to investigate, over
3 years, the progress of three groups of instrument students using PCATDs for 5, 10, and 15 hours of
training in an instrument training course. A control group would receive all training in an airplane. The
use of flight training devices will not be included in the study. An additional study is now examining
the training value of PCATDs for maintaining pilots’ skills and for substituting these devices for
aircraft experience in maintaining instrument skills. According to a University of Illinois researcher,
the airplane performance of 100 current instrument pilots will be tested and compared across four
groups: A control group will receive no practice in instrument flight during a 6-month period; an
airplane group will perform the six required approaches and holding patterns in the aircraft; a PCATD
group will perform the same maneuvers in a PCATD; and a flight training device group will perform
the same maneuvers in a flight training device. The performance of pilots on the final instrument
proficiency check ride at the end of the 6-month experimental period will be compared with the
performance on the initial instrument proficiencies check ride. Pilots may not receive recency credit
for their practice on PCATDs. To meet FAA recency regulations toward maintaining current
instrument skills, pilots must make at least six instrument approaches, holding procedures, and
intercepting and tracking courses every 6 months. These maneuvers must be performed under either
simulated or actual flight conditions; currently, flight training devices may be used to perform all six
instrument currency approaches, but PCATDs may not.



Page 9                   GAO/RCED-99-143 Personal Computer-Based Aviation Training Device
                            B-280735




                            The two FAA-commissioned studies by the University of Illinois and
Research and Experts        Embry-Riddle Aeronautical University are the most complete controlled
Support Training            studies to date on the training effectiveness of computer-based devices.
Value of PCATDs, but        Despite methodological limitations, the studies show that (1) the use of
                            computer-based devices may modestly reduce the training time spent in
Devices’ Impact on          the airplane (Illinois) and (2) the training effects of computer-based
Long-Term Safety Is         devices and a previously approved flight training device may not differ
                            greatly (Embry-Riddle). The additional studies we reviewed also
Unknown                     supported the use of computer-based devices. None of these studies
                            examined the issue of how many hours of training on a computer-based
                            device should be allowed in an instrument training curriculum. Academic
                            and other flight training experts generally believed that computer-based
                            devices offer training value to pilots obtaining an instrument rating.

                            Since PCATDs have only recently been approved and few pilots have used
                            them in training, their long-term risks and benefits are not yet known.
                            While several studies examined the training effectiveness of
                            computer-based devices, none directly addressed their effects on general
                            aviation safety. Safety experts we interviewed did not believe that the use
                            of computer-based devices and the granting of credit hours pose a threat
                            to general aviation safety by adversely affecting the skills and abilities of
                            pilots who receive instrument training using the devices. However, several
                            other issues that could affect the long-term ability of PCATD-trained pilots
                            to fly safely have been raised.


Research Generally          The Illinois study showed that the use of computer-based devices in an
Supports Use of PCATDs in   instrument course may modestly reduce the training time spent in the
Training                    airplane—26.5 hours on the device during an instrument course reduced
                            the amount of airplane training time needed by 3.9 hours. (See. fig. 2.)




                            Page 10           GAO/RCED-99-143 Personal Computer-Based Aviation Training Device
                                          B-280735




Figure 2: University of Illinois Study:
Hours to Proficiency in an Airplane       Hours




                                                     Hours using PC

                                                     Hours in airplane


                                          Notes: N = Number of students trained.

                                          The airplane used was a Beechcraft Sports/Sundowner. The computer-based device was an
                                          MDM FS-100, not FAA-approved as a PCATD.

                                          Source: GAO’s analysis of University of Illinois data.




                                          The Embry-Riddle study provided a preliminary indication that there may
                                          not be very large differences between the training effects of the previously
                                          approved flight training devices and the recently approved
                                          computer-based devices. The training trials and time needed to reach
                                          proficiency for students using two different types of computer-based
                                          devices and for students using a previously approved flight training device
                                          are shown in figures 3 and 4. (See app. II for a more detailed summary of
                                          the two FAA-sponsored studies.)




                                          Page 11                  GAO/RCED-99-143 Personal Computer-Based Aviation Training Device
                                          B-280735




Figure 3: Embry-Riddle Study: Trials to
Proficiency in an Airplane                Trials




                                          Training device used


                                          Notes: N = Number of students trained.

                                          The airplane used was the Mooney 20J. The Elite PC was a computer-based device, an
                                          FAA-approved PCATD. The IFT PC was a computer-based device not approved by FAA as a
                                          PCATD. The Frasca FTD is a flight-training device approved by FAA.

                                          Source: GAO’s analysis of Embry-Riddle Aeronautical University data.




                                          Page 12                GAO/RCED-99-143 Personal Computer-Based Aviation Training Device
                                      B-280735




Figure 4: Embry-Riddle Study: Hours
to Proficiency in an Airplane         Hours




                                      Training device used


                                      Notes: N = Number of students trained.

                                      The airplane used was the Mooney 20J. The Elite PC was a computer-based device, an
                                      FAA-approved PCATD. The IFT PC was a computer-based device not approved by FAA as a
                                      PCATD. The Frasca FTD is a flight-training device approved by FAA.

                                      Source: GAO’s analysis of Embry-Riddle Aeronautical University data.




                                      The additional studies we reviewed also found some value in using
                                      computer-based devices. Three studies found computer-based devices
                                      helped train beginning pilots.7 An additional study found that performance
                                      in an airplane test was better for a group of students trained on




                                      7
                                       Gustavo Ortiz, “Effectiveness of PC-Based Flight Simulation,” The International Journal of
                                      Aviation Psychology 4 (3), p. 285, 1994; Kerry A Dennis and Don Harris, “Computer-Based
                                      Simulation as an Adjunct to Ab Initio Flight Training,” The International Journal of Aviation
                                      Psychology 8 (3), pp. 261-276, 1998; Jefferson M. Koonce, Steven L. Moore, and Charles J. Benton,
                                      “Initial Validation of a Basic Flight Instruction Tutoring System (BFITS),” Proceedings of the Eighth
                                      International Symposium on Aviation Psychology, Columbus, Ohio, 1995.



                                      Page 13                  GAO/RCED-99-143 Personal Computer-Based Aviation Training Device
                             B-280735




                             computer-based devices than for a group trained on flight training devices.8



Experts See Training Value   The academic and flight training experts as well as the flight instructors
in Computer-Based            we interviewed generally believed that PCATDs offer training value to
Devices                      instrument students. Experts told us that training devices, if used properly,
                             have the potential to be more effective than airplane training for certain
                             uses; furthermore, training experts generally agree that a low-cost
                             simulation device may be justified economically even if the training it
                             provides is less effective than the training provided in an airplane. In
                             addition, commercial airline training and military research support the
                             value of computer-based training for certain training purposes. Several
                             experts we interviewed believed that computer-based devices would
                             become more effective as their technology continues to improve and as
                             features are added to enhance their training effectiveness.

                             Some experts believe that a variety of training equipment, including
                             computer-based devices, can offer effective training if used properly.
                             PCATDs are usually seen by experts as most appropriate for introducing
                             equipment and procedures and for practicing certain maneuvers, flight
                             scenarios, and emergency procedures. Flight instructors in one survey
                             believed that computer-based devices were effective for teaching most
                             tasks in an instrument course.9

                             Flight training experts at a major airline’s training center told us that they
                             and other airlines use a variety of training equipment in their operations,


                             8
                              Sybil I. Phillips, Charles L. Hulin, and Paul J. Lamermayer, “Uses of Part-Task Trainers in Instrument
                             Flight Training,” University of Illinois, Proceedings of the Seventh International Symposium on
                             Aviation Psychology, Columbus, Ohio, 1993.
                             9
                              William F. Moroney, Steven Hampton, and David W. Biers, “Considerations in the Design and Use of
                             Personal Computer-Based Aircraft Training Devices (PCATDs) for Instrument Flight Training: A
                             Survey of Instructors,” University of Dayton, Ohio, and Embry-Riddle Aeronautical University,
                             Daytona Beach, Florida, Proceedings of the Ninth International Symposium on Aviation Psychology,
                             Columbus, Ohio, 1997. According to another study (Brian K. Rogers, Capt., U.S. Air Force.
                             “Microcomputer-Based Instrument Flight Simulation: Undergraduate Pilot Training Student Attitude
                             Assessment,” December 1991, Human Resources Directorate, Aircrew Training Research Division,
                             Williams Air Force Base, Arizona, AL-TR-1991-0039), surveyed students believe that computer-based
                             devices have some training value. Almost all of the of U.S. Air Force flight students surveyed planned
                             to use computer-based instrument simulators to replace at least some of their “chair-flying,” or
                             rehearsing procedures from a chair placed in front of a life-size photo of the airplane’s cockpit. Over
                             half said they would replace at least 75 percent of their chair-flying with computer-based simulation.
                             The same study found increased student interest in using computer-based simulation as instrument
                             training was expanded to include multiple procedures in real-time practice. Another study (D. Gopher,
                             M. Weil, and T. Bareket, “Transfer of skill from a computer game trainer to flight,” Human Factors, 36
                             (3) (1994), pp. 387-405) found that military flight students who played a complicated computer game
                             performed better in later airplane training.



                             Page 14                  GAO/RCED-99-143 Personal Computer-Based Aviation Training Device
                           B-280735




                           ranging from $20 million full-motion simulators to simple personal
                           computer-based training, depending on whether their goal is to introduce a
                           pilot to a new procedure or piece of equipment, to practice complete
                           simulated flights, or to serve an intermediate purpose. For example, these
                           airline training officials believed that simple personal computer equipment
                           can introduce a pilot to a new navigation system or component more
                           effectively than a full-motion simulator with its many competing demands
                           for a pilot’s attention.


Safety and Academic        According to safety experts from the National Transportation Safety Board
Experts Generally Do Not   and from an industry safety association, the use of computer-based
Believe PCATDs Pose a      devices and the granting of credit hours do not pose a threat to general
                           aviation safety by adversely affecting the skills and abilities of pilots who
Safety Risk                receive instrument training using the devices. In general, academic and
                           other flight training experts believe that the devices offer training value to
                           pilots who want to obtain an instrument rating. According to these
                           experts, good instrument and navigational skills are vital in avoiding bad
                           weather and surviving it when necessary; two experts we interviewed
                           believe that the low cost and wide availability of PCATDs could help
                           maintain or increase the general level of pilots’ instrument skills, which
                           are critical to safe flying.


Long-Term Safety Impact    We found no empirical evidence on the long-term safety impact of
of PCATDs Is Unknown       computer-based devices, their potential safety benefits, or the long-term
                           safety impact or benefits of the currently approved flight training devices.
                           Many of the experts we interviewed believed either that computer-based
                           devices were not likely to reduce aviation safety or that they offer
                           long-term safety benefits. However, several experts noted that the
                           potential for long-term safety risks from computer-based devices used in
                           training remains a possibility. Objections to their use for credit towards
                           the training hours needed for an instrument rating generally revolve
                           around whether using a computer-based device in instrument training is
                           more likely to reduce or increase an instrument pilot’s margin of safety.
                           Several issues were raised in our interviews with experts concerning PCATD
                           safety. Among these were potential risks from the lack of physical
                           similarity between the device and the airplane and associated issues,
                           possible benefits to training from their use, and the lack of research
                           showing that one hour on a training device equals one hour of training in
                           an airplane. Some of these issues appear to support the value of PCATDs;
                           others raise questions about their value; and the effects of other issues on



                           Page 15           GAO/RCED-99-143 Personal Computer-Based Aviation Training Device
              B-280735




              the value of PCATDs are unclear. Although not fully resolved, these issues
              may be relevant to future research. (See app. III for a summary of these
              issues.)

              FAA does not currently track how many hours an instrument pilot trains on
              a PCATD. However, FAA is now computerizing this information and, for
              unrelated reasons, plans to revise the form on which pilots apply for the
              instrument and other ratings. FAA’s application for the instrument rating
              could easily and inexpensively be changed to include a box on the types of
              training devices a pilot has used, particularly since the form (FAA Form
              8710-1) will soon be revised and its data computerized. With information
              on the training devices used by a particular pilot, along with other data
              that FAA now maintains or will soon maintain electronically, researchers
              will be better able to determine, within a few years, the safety of pilots
              trained using computer-based devices. These data will enable researchers
              to link accidents or incidents with the type of instrument training the
              pilots had received. While this type of examination would present some
              statistical challenges, experts we interviewed said that it would be
              possible to identify any large safety risks posed by computer-based
              devices.


              Commercial aviation safety depends on many factors, including the
Conclusions   training of general aviation pilots, who sometimes share common airspace
              and airports with the flying public. Because most accidents in general
              aviation are caused by pilot error, and many accidents occur when pilots
              without an instrument rating encounter bad weather or poor visibility and
              unexpectedly have to rely on their instruments, effective training that
              teaches, reinforces, and maintains safe instrument flying skills is vital. To
              the extent that the use of computer-based devices may be of training value
              to instrument pilots—which research supports and experts believe—the
              devices do not pose identifiable safety risks.

              The low cost and wide availability of computer-based devices could
              improve the general level of instrument training among pilots, according to
              some experts, and help to maintain or improve instrument pilots’ skills,
              especially for students and pilots who would not otherwise have access to
              a training device. Several experts told us that FAA’s check rides, which are
              required for a pilot to obtain an instrument rating, may not be sufficient
              indicators of a student’s training, capabilities, and ability to fly safely.
              However, we found no studies on the link between a pilot’s training
              equipment or flying proficiency at the time of training and that pilot’s



              Page 16           GAO/RCED-99-143 Personal Computer-Based Aviation Training Device
                  B-280735




                  long-term ability to fly safely. Collecting information on pilots’ training
                  equipment and safety records would permit research in the future and
                  could help to uncover any safety problems related to the use of
                  computer-based and other training devices.


                  As a first step toward determining whether computer-based devices pose
Recommendations   safety risks to general aviation in relation to other training methods, the
                  Secretary of Transportation should direct the FAA Administrator to collect
                  information from pilot applications for instrument ratings on how many
                  hours students trained on PCATDs and flight training devices. This
                  information can be used by FAA and others to study the relationship
                  between the use of training devices and safety. FAA’s revision to its rating
                  application form allows the additional information to be obtained at
                  minimal cost.


                  We provided a draft copy of this report to the Office of the Secretary of
Agency Comments   Transportation and the Federal Aviation Administration (FAA) for review
                  and comment. We discussed the report with FAA officials, including the
                  Deputy Associate Administrator for Regulation and Certification. They
                  generally agreed with the draft report. FAA officials also indicated that they
                  concurred with and would implement our recommendation as soon as
                  possible. They also provided technical corrections, which we incorporated
                  into the report as appropriate.


                  We reviewed FAA’s decision to allow the use of PCATDs by reviewing the
Scope and         data the agency used to support its decision and interviewing a variety of
Methodology       FAA and aviation industry officials. We also reviewed research on the use
                  of computer-based devices in training and interviewed a variety of experts
                  on the potential safety impacts of their use. Appendix I provides our
                  detailed scope and methodology.

                  We conducted our review from June 1998 through June 1999 in
                  accordance with generally accepted government auditing standards.


                  As arranged with your offices, unless you announce its contents earlier,
                  we plan no further distribution of this report until 7 days after the date of
                  this report. At that time, we will make copies available to the Secretary of




                  Page 17            GAO/RCED-99-143 Personal Computer-Based Aviation Training Device
B-280735




Transportation and the Administrator, FAA. We will also make copies
available to others on request.

If you have any questions about this report, please contact me at
(202) 512-3650. Major contributors to this report were Karen Bracey, David
Ehrlich, and John Rose.




Gerald L. Dillingham
Associate Director, Transportation Issues




Page 18          GAO/RCED-99-143 Personal Computer-Based Aviation Training Device
Page 19   GAO/RCED-99-143 Personal Computer-Based Aviation Training Device
Contents



Letter                                                                                                  1


Appendix I                                                                                             22

Scope and
Methodology
Appendix II                                                                                            25
                        University of Illinois Study                                                   25
Summaries of Two        Embry-Riddle Aeronautical University Study                                     26
Major FAA-Sponsored
Studies
Appendix III                                                                                           28
                        Views on the Lack of Physical Similarity Between PCATDs and                    28
Other Issues Related      Aircraft Instrument Panels and Controls
to PCATD Safety
Selected Bibliography                                                                                  41


Figures                 Figure 1: Personal Computer-Based Aviation Training Device                      5
                        Figure 2: University of Illinois Study: Hours to Proficiency in an             11
                          Airplane
                        Figure 3: Embry-Riddle Study: Trials to Proficiency in an Airplane             12
                        Figure 4: Embry-Riddle Study: Hours to Proficiency in an                       13
                          Airplane
                        Figure III.1: Approved Flight Training Device                                  29
                        Figure III.2: Instrument Panel Photo of Cessna 172                             32
                        Figure III.3: PCATD Depiction of Cessna 172 Instruments                        33
                        Figure III.4: Flight Path Screen                                               35
                        Figure III.5: Screen to Set Failures                                           36
                        Figure III.6: Screen to Set Weather Conditions                                 37
                        Figure III.7: Example of Grandfathered Flight Training Device                  39




                        Abbreviations

                        CAMI       Civil Aeromedical Institute
                        FAA        Federal Aviation Administration
                        FTD        flight training device
                        PCATD      personal computer-based aviation training device


                        Page 20           GAO/RCED-99-143 Personal Computer-Based Aviation Training Device
Page 21   GAO/RCED-99-143 Personal Computer-Based Aviation Training Device
Appendix I

Scope and Methodology


             To examine the Federal Aviation Administration’s (FAA) decision allowing
             the use of personal computer-based aviation training devices (PCATD), we
             reviewed the data the agency used to support its decision to issue
             Advisory Circular 61-126, reviewed other studies available on the topic,
             and interviewed a variety of experts within and outside government. In
             particular, we reviewed the supporting data FAA used in making its
             decision, including a 1994 study by Embry-Riddle Aeronautical University
             and a 1996 study by the University of Illinois on the effectiveness of PCATD
             training. We reviewed FAA’s files, including archived e-mail messages, a
             chronology of events, and supporting information and documents. We met
             with FAA officials from Flight Standards, both in Washington, D.C., and at
             an Illinois Flight Standards District Office, as well as with FAA officials
             from Human Factors, the Office of Aviation Medicine, the Civil
             Aeromedical Institute (CAMI) in Oklahoma City, and the National Simulator
             Program in Atlanta.

             We asked several recognized academic and training experts to review our
             plans at several points during our research. Each of these experts has
             extensive academic, research, and training experience in aviation,
             including the fields of aviation psychology and human factors; their views
             are theirs alone rather than those of their institutions. These experts were
             Dr. Barry Strauch of the National Transportation Safety Board; Dr. Dee
             Andrews of the U.S. Air Force Research Laboratory, Mesa, Arizona; and
             Professor (Emeritus) Stanley Roscoe of the University of Illinois at
             Urbana-Champaign and New Mexico State University.

             In addition, we interviewed flight training, aviation psychology, human
             factors, military research, and aviation safety experts in academia and the
             military, as well as at the National Transportation Safety Board and the
             National Aeronautics and Space Administration. We interviewed flight
             instructors, association and airline officials, the manufacturers of each of
             the approved PCATDs, and the manufacturers of flight training devices and
             control equipment.

             In particular, we interviewed representatives of the following
             organizations: the National Association of Flight Instructors; the National
             Air Transport Association; the University Aviation Association; the
             Airplane Owners and Pilots Association and its affiliated Air Safety
             Foundation; two flight schools in the Chicago area; two flight schools in
             the Washington, D.C., area; and the chief flight instructors of the
             University of Illinois and Embry-Riddle Aeronautical University at Daytona
             Beach, and other selected flight instructors. Among the equipment



             Page 22           GAO/RCED-99-143 Personal Computer-Based Aviation Training Device
Appendix I
Scope and Methodology




manufacturers and distributors we interviewed were Aviation Supplies &
Academics, Inc.; Aviation Teachware Technologies; Jeppesen Sanderson;
and Frasca International, Inc. Among the academic experts we contacted,
in addition to our advisory panel, were the authors of both the University
of Illinois and Embry-Riddle Aeronautical University studies. At FAA, we
spoke to officials in Flight Standards; the National Simulator Center
(Atlanta); the Civil Aeromedical Institute (Oklahoma City); Human Factors
technical scientists (Washington, D.C.); and FAA’s Flight Standards District
Office (West Chicago, Illinois). We also spoke to representatives of the
U.S. Air Force, the U.S. Navy, the National Transportation Safety Board,
and the National Aeronautics and Space Administration (Ames,
California).

To identify other studies relevant to FAA’s decision, we obtained articles
cited in recent studies. We also conducted additional literature searches of
the collections at two major aviation libraries and a computerized
literature search using several bibliographic databases and including the
following keywords and phrases: personal computer aviation training
device, PCATD, accident/incident, flight training device, ground training
device, part task trainers, aviation safety, pilot training, air safety.10 Our
computerized literature search identified about 700 studies and articles. Of
these, we selected 68 dealing with personal computers, simulation issues,
or aviation safety, including all the citations that appeared to involve,
discuss, or present empirical research in these areas. Thirty-nine abstracts
were further selected as relevant enough to obtain full text copies,
including all those that appeared to involve empirical research on personal
computer-based flight training. From all our sources, including electronic
and nonelectronic literature searches, we found eight studies involving
computer-based device effectiveness, and four of these met all of the
following criteria: (1) examined computer-based flight simulation
effectiveness; (2) compared computer-based device training to
airplane-only training performance or to training with another approved
training device; (3) had been conducted using an experimental design or
survey, including random assignment; and (4) used data from an
experiment not summarized elsewhere in our report (i.e., the two studies
cited by FAA in its decision—one at Embry-Riddle Aeronautical University
and one at the University of Illinois). In addition, we reviewed studies and



10
 The databases searched included the National Technical Information Service and Transportation
Research Information Services; DIALOG, from which the following databases were searched:
Dissertation Abstracts Online, GPO Monthly Catalog, PsychINFO, IAC Aerospace/Defense Markets,
Aerospace Database, McGraw-Hill Publications, Periodical Abstracts Plus text, DIALOG Defense
Newsletters, Business Dateline, IAC Newsletter Database; and the NEXIS database.



Page 23                GAO/RCED-99-143 Personal Computer-Based Aviation Training Device
Appendix I
Scope and Methodology




articles on general aviation flight safety that did not directly mention
either computer-based simulation or training.

Many articles in this field are published in journals with limited circulation
and in the proceedings of civilian and military conferences. Though these
can be difficult to identify from conventional bibliographic databases, we
attempted through these combined methods to obtain all those studies
examining computer-based flight training that used empirical methods.

We conducted our review from June 1998 through June 1999 in
accordance with generally accepted government auditing standards.




Page 24            GAO/RCED-99-143 Personal Computer-Based Aviation Training Device
Appendix II

Summaries of Two Major FAA-Sponsored
Studies

                         This appendix summarizes the studies by the University of Illinois and the
                         Embry-Riddle Aeronautical University.


                         In the University of Illinois study,11 researchers compared the performance
University of Illinois   of students who received some training on a PCATD before their training in
Study                    an airplane with the performance of a group of students trained entirely in
                         an airplane. University of Illinois aviation students who were already
                         private pilots and had no previous instrument coursework were randomly
                         assigned to one of two groups for their instrument flight-training course.
                         After this study’s coursework, passing students took FAA check rides to
                         attempt to obtain their instrument rating. Performance measures were
                         available for 74 students; roughly half the students were in each group.
                         Students were tested to demonstrate proficiency, and the time and number
                         of trials were recorded for each student. The computer-based device used
                         was Model FS-100 by MDM, a model not currently approved for use as a
                         PCATD by FAA toward flight training credit hours; the airplane used was
                         Beechcraft Sports/Sundowner, which has a single engine, a fixed-pitch
                         propeller, and fixed landing gear.

                         Students’ performance in the airplane was not substantially different for
                         those trained partly using a computer-based device and those trained
                         solely in an airplane. However, the data showed small, but statistically
                         significant, savings in the total airplane time needed to reach proficiency
                         in the airplane by students trained on the PCATD device. Students using this
                         device spent, on average, 3.9 fewer hours in the airplane than those who
                         took their entire training in the airplane. Airplane-only students averaged
                         51.3 total hours to complete the course, while students using the PCATD
                         averaged 47.4 hours in an airplane and 26.5 on the device.

                         This study was not designed to determine the amount of loggable time FAA
                         should grant for PCATD use, according to its authors. It did not include a
                         student group trained on a currently approved flight training device. Such
                         a comparison group is of interest because the 10 hours currently permitted
                         for PCATD instrument training may be substituted for part of the 15 to 20
                         hours that can be earned using a flight training device. The study’s authors
                         believe that in an actual instrument course, where experimental controls
                         would not require the use of the PCATD for all course lessons, the 3.9 hour
                         savings of time in the airplane could be achieved with fewer PCATD hours

                         11
                          Henry L Taylor, Gavan Lintern, Charles L. Hulin, Donald Talleau, Tom Emanuel, and Sybil Phillips;
                         “Transfer of Training Effectiveness of Personal Computer-Based Aviation Training Devices,”
                         University of Illinois; November 1996, Final Technical Report ARL-96-3-/FAA-96-2, prepared for the
                         Federal Aviation Administration, Oklahoma City, OK, Contract DTFA-94-G-044



                         Page 25                 GAO/RCED-99-143 Personal Computer-Based Aviation Training Device
                   Appendix II
                   Summaries of Two Major FAA-Sponsored
                   Studies




                   by targeting the use of the device to those early course lessons where
                   training transfer is greatest.

                   The study does not support a conclusion that a “negative transfer”—or
                   learning interference caused by using PCATDs—occurred when the students
                   trained on the personal computer device switched to the airplane. In the
                   few instances in which this group of students needed more trials to
                   demonstrate proficiency in the airplane than the group without PCATD
                   training, the differences were so small that they were not statistically
                   significant. Differences of the magnitude observed could occur even if the
                   two groups of students had been given exactly the same training. While the
                   study does not support a conclusion that negative transfer occurred on
                   some tasks, it also does not rule out the possibility that it did occur. Small
                   amounts of negative transfer in an individual task could have existed
                   without being detected by statistical tests.


                   The Embry-Riddle study measured the flight performance of students
Embry-Riddle       trained using PCATDs and compared their performance with that of
Aeronautical       students trained using a previously approved flight training device.12 All
University Study   the students had previously taken navigation and instrument coursework.
                   Embry-Riddle students who volunteered for the study were randomly
                   assigned to receive ground-based training using one of two PCATDs or using
                   the flight training devices. Students used the assigned ground-training
                   devices during the first part of their training and continued their
                   instrument flight training in an airplane. Instructors were randomly
                   assigned to students and taught students in each of the three groups. In the
                   airplane, the performances of the three groups of students were measured
                   by the total amount of time and the total number of trials students
                   required to demonstrate proficiency in each of eight maneuvers.
                   Performance measures were available for a total of 66 students. The
                   PCATDs used were the Elite (20 students finished) and the IFT Pro 5.1 (26
                   students finished); both devices had 15-inch screens, 486/66MHz
                   computers, and similar control features. The flight training device used
                   was the Frasca 141 (20 students finished). The airplane used was a
                   Mooney 20J single-engine plane.

                   The study provided a preliminary indication that there may not be very
                   large differences in the ground flight training provided by the approved

                   12
                    Steven Hampton, William Moroney, Tom Kirton, and David W. Biers, “The Use of Personal
                   Computer-Based Training Devices in Teaching Instrument Flying,” Embry-Riddle Aeronautical
                   University, June 15, 1994. Prepared for the Federal Aviation Administration, CAAR 15471931, Grant No.
                   92-G-0015.



                   Page 26                 GAO/RCED-99-143 Personal Computer-Based Aviation Training Device
Appendix II
Summaries of Two Major FAA-Sponsored
Studies




flight training devices and the two PCATDs studied. No statistically
significant differences were found in the airplane performance of the
groups trained on the three devices, as measured by trials and hours to
meet FAA’s established practical test standards.

The study did not and was not intended to address the issue of how many
hours of training on PCATDs FAA should grant for an instrument rating,
according to one of the study’s authors. The study also did not show how
much, if any, training received on each of the ground-based devices was
transferred to the airplane. The performance of a group of students who
received all their training in an airplane would be required before an
estimate of how much less time in an airplane students receiving training
on a PCATD or flight training device might need. Because no empirical
studies have been conducted on the effectiveness of training on a flight
training device compared with airplane-only training for instrument
students, this study’s comparative data cannot be used to conclude that
PCATDs provide effective instrument training.




Page 27             GAO/RCED-99-143 Personal Computer-Based Aviation Training Device
Appendix III

Other Issues Related to PCATD Safety


                       We identified a number of other issues from interviews with safety,
                       industry, and academic experts that could affect the long-term use of
                       computer-based devices. These issues are the lack of the devices’ physical
                       similarity to actual airplanes and the potential for associated problems
                       from negative learning from simulation—that is, a pilot’s reverting to
                       incorrectly learned behaviors in an emergency or under stress—and the
                       possibility that the smaller-sized instruments on the devices could
                       interfere with a pilot’s instrument scan. Possible benefits from PCATDs
                       include their availability and use in teaching concepts and procedures,
                       such as the flexibility they provide by permitting a pilot to practice on a
                       particular type of airplane, to see a graphic display of performance, and to
                       fly under varying scenarios of instrument failure, system failure, and
                       weather conditions.

                       Several simulation experts believe that credit hours should not be granted
                       for PCATD use, that the two major studies do not show the training
                       effectiveness of either PCATDs or flight training devices, and that there is no
                       evidence that one hour in any training device equals one hour in an
                       airplane, particularly for the older, grandfathered flight training devices.


                       One of the main concerns expressed by those opposed to granting credit
Views on the Lack of   for instrument training time on PCATDs is that PCATDs do not have sufficient
Physical Similarity    similarity, or physical fidelity, to an airplane’s instrument panel and
Between PCATDs and     controls.13 For example, PCATDs have display screens showing instruments
                       that are smaller than they actually are in the cockpit, and the placement of
Aircraft Instrument    some PCATD switches and dials represented on the display screen are in a
Panels and Controls    physically different position than in the aircraft—alongside or below a
                       desktop computer monitor. Flight training devices, in contrast, can include
                       more instruments—often actual aircraft instruments—that can be
                       mounted on a dashboard panel larger than a computer screen, permitting
                       greater physical similarity to an aircraft. On the other hand, flight training
                       devices represent a standard aircraft configuration, while PCATD software
                       can be easily changed to represent a specific aircraft. Figure III.1 shows an
                       approved flight training device.




                       13
                         The literature discusses at least three different kinds of fidelity, including physical fidelity—whether
                       the simulation “looks right;” functional fidelity—whether the simulation “acts right;” and
                       psychological fidelity—whether the simulation “feels right.”



                       Page 28                  GAO/RCED-99-143 Personal Computer-Based Aviation Training Device
                                         Appendix III
                                         Other Issues Related to PCATD Safety




Figure III.1: Approved Flight Training
Device




                                         Note: This is a previously approved flight training device with an optional visual system (Frasca
                                         131).

                                         Source: Frasca International.




                                         Page 29                 GAO/RCED-99-143 Personal Computer-Based Aviation Training Device
Appendix III
Other Issues Related to PCATD Safety




The potential impact of these differences between PCATDs and flight
training devices on pilot training is not clear. Some experts and
representatives of the flight training device industry believe that devices
that are less similar to the airplane and have lower physical fidelity could
interfere with learning. Some simulation experts believe that the more the
simulator looks, feels, and “flies” like the airplane, the greater its training
value will be, particularly for experienced pilots. However, other experts
and many research studies suggest that high physical fidelity is not
essential for a device to have training value; that students using
low-fidelity devices often perform as well as those using more complex,
higher-fidelity devices; and that the effectiveness of a training device
depends on how the device is used in a training curriculum.14

Negative transfer—learning that can interfere with the performance of a
task rather than improve it—can occur when a familiar instrument or knob
is placed in a dramatically different position in a training device than in an
aircraft. One expert on aviation psychology told us that negative transfer
from PCATDs would be unlikely but possible. A pilot’s switching from a
training device to an airplane, or from one airplane to another, according
to interviews with aviation psychologists, would rarely involve locating
control yokes, knobs, or switches that are completely reversed. According
to one expert, however, if a pilot trains in a plane with the retractable
landing gear switch on the left side of the cockpit and the flap switch on
the right and then flies another plane with the landing gear switch on the
right and the flap switch on the left, negative transfer of learning could
occur if a pilot were to confuse the two switches.

In general, the academic, private, military, and airline flight training
experts we interviewed did not believe that the issue of PCATDs’ similarity
to the airplane presented safety problems. Several safety experts noted
that previously approved flight training devices do not typically represent
one particular type of aircraft and often include instrument panels that
differ from those on the plane in which the student will train and later fly;
students will have to adjust from the training device to a somewhat
different airplane. While familiarity with a particular type of plane is an
important safety consideration, few students will fly only one type of plane
and therefore will have to adjust, using training devices, to different



14
 Gavan Lintern, S.N. Roscoe, J.M. Koonce, and L.D. Segal, ”Transfer of Landing Skill in Beginning
Flight Training,” Human Factors 32, (1990), pp. 319-327; Gavan Lintern, Henry L. Taylor, Jefferson M.
Koonce, and Donald Talleur, “An Incremental Transfer Study of Scene Detail and Field of View Effects
on Beginning Flight Training,” Proceedings of the Eighth International Symposium on Aviation
Psychology, Columbus, Ohio, 1995, p. 737.



Page 30                 GAO/RCED-99-143 Personal Computer-Based Aviation Training Device
Appendix III
Other Issues Related to PCATD Safety




instrument displays, a different feel to the controls, and different
performance characteristics.

Another potential danger could occur when a pilot, in an emergency,
automatically reverts to incorrectly learned responses, skills, or
procedures. This stressful reversion, according to some industry
representatives, could be the result of the lower physical fidelity of PCATDs.
Other experts dispute this, noting that instrument students are already
pilots licensed to fly in good visibility conditions and already have some
experience using airplane instrumentation; one expert noted that incorrect
and dangerous initial instruction can take place in an airplane as well as
while using a PCATD. Another expert said that quality instruction is more
important than the type of training device, but this type of problem would
be difficult to detect.

Several experts were also concerned that the smaller size of PCATD
instruments—usually about three-fourths actual size—coupled with the
smaller size of the PCATD computer monitor screen, could affect the quality
of a pilot’s instrument scan—a critical skill to retain constant spatial
orientation by using instruments alone.15 In poor visibility, a pilot must
quickly scan the correct instruments that are most important to
completing various maneuvers, as well as to maintain level flight.
Instrument scan may also be impaired if PCATD-displayed instruments
suffer from lower fidelity than flight training devices, some PCATD critics
believe. Figures III.2 and III.3 show an actual instrument panel and the
PCATD version of that panel.




15
  Spatial orientation is knowing the position of the plane in space and its speed, direction, and up,
down, and sideways orientation. A blindfolded passenger in a plane—even an experienced
pilot–quickly loses the ability to accurately sense whether a plane is climbing, descending, or turning,
in part because of the confusing multiple sensations of acceleration, deceleration, and turning from
vestibular and other cues. G-forces and even a turning of the head during a turn or climb can cause
equally misleading illusions. A pilot flying relying on visual cues can be tragically mistaken by subtle
optical illusions under certain conditions that might be prevented by the better use of instruments to
confirm the plane’s situation.



Page 31                  GAO/RCED-99-143 Personal Computer-Based Aviation Training Device
                                          Appendix III
                                          Other Issues Related to PCATD Safety




Figure III.2: Instrument Panel Photo of
Cessna 172




                                          Source: The Cessna Aircraft Company.




                                          Page 32               GAO/RCED-99-143 Personal Computer-Based Aviation Training Device
                                   Appendix III
                                   Other Issues Related to PCATD Safety




Figure III.3: PCATD Depiction of
Cessna 172 Instruments




                                   Source: Aviation Teachware Technologies




                                   While instrument and panel size could be potential problems for a pilot’s
                                   instrument scan, experts responded to this potential problem in several
                                   ways. Several noted that a pilot’s instrument scan must be altered to read
                                   different airplanes’ instrument panels and so may be aided by practice on
                                   a PCATD that resembles the particular plane he or she flies because primary
                                   instruments would be correctly spaced although not full-sized. A flight
                                   instructor noted that larger computer monitors are becoming less
                                   expensive and that larger screens can increase the size of a PCATD’s
                                   instrument display. A safety expert noted that whether an instrument is an
                                   actual airplane instrument or displayed on a video screen is not important.
                                   Other experts say that clarity on a video screen can be very good—most
                                   commercial airliners now display instruments on video screens in “glass
                                   cockpits.” Another expert said that while very few general aviation




                                   Page 33               GAO/RCED-99-143 Personal Computer-Based Aviation Training Device
                          Appendix III
                          Other Issues Related to PCATD Safety




                          aircraft now have glass cockpits, they will become more common in the
                          future.


Views on the Advantages   Experts we interviewed saw the main value of PCATDs in teaching
of PCATDs                 procedures and concepts, rather than the complete set of skills needed to
                          fly. Many experts and flight instructors believe that isolating a single
                          procedure on a training device can help a student focus on that lesson and
                          learn more effectively, particularly when introduced to a new topic,
                          without the distractions of simultaneously flying the airplane.

                          In general, the experts we interviewed believed that the potential safety
                          advantages of PCATDs outweigh their potential risks. Among these possible
                          benefits are the flexibility of PCATD software to mimic the instrument
                          configuration and performance of a variety of airplanes; permitting the
                          pilot to fly a half-dozen approaches in an hour compared with perhaps one
                          in an airplane; allowing the pilot to see and save a chart of a flight as a
                          learning tool; and allowing the simulation of systems failures, specific
                          instrument failures, and a variety of weather conditions, such as wind
                          speed and level of visibility. Figure III.4 shows a PCATD diagram of a flight
                          path, figure III.5 shows a PCATD screen to set instrument failures and figure
                          III.6 shows a PCATD screen to set weather conditions.




                          Page 34             GAO/RCED-99-143 Personal Computer-Based Aviation Training Device
                                   Appendix III
                                   Other Issues Related to PCATD Safety




Figure III.4: Flight Path Screen




                                   Source: Aviation Teachware Technologies.




                                   Page 35               GAO/RCED-99-143 Personal Computer-Based Aviation Training Device
                                       Appendix III
                                       Other Issues Related to PCATD Safety




Figure III.5: Screen to Set Failures




                                       Source: Aviation Supplies & Academics, Inc.




                                       Page 36                GAO/RCED-99-143 Personal Computer-Based Aviation Training Device
                                      Appendix III
                                      Other Issues Related to PCATD Safety




Figure III.6: Screen to Set Weather
Conditions




                                      Source: Aviation Teachware Technologies.




                                      Some safety experts also see the wider availability and use of relatively
                                      inexpensive PCATDs as a benefit, since most instrument students are not
                                      likely to have easy access to a flight training device.16 The wider use of
                                      PCATDs for continuing training could potentially improve the overall level
                                      of general aviation pilots’ instrument skills and judgment, which in turn
                                      could improve pilot safety.


Views on Granting Training            According to several experts, PCATDs have a role in training but that
Credit for PCATDs                     students using PCATDs should not be granted credit hours for instrument
                                      coursework. These experts believe that neither of the two major studies
                                      showed that PCATDs had achieved a level of training effectiveness so that
                                      one hour of device training equals one hour in the airplane; however, FAA

                                      16
                                       According to PCATD manufacturers, many PCATDs are sold to individuals either for practice at
                                      home or possibily for individual Part 61 instruction.



                                      Page 37                GAO/RCED-99-143 Personal Computer-Based Aviation Training Device
Appendix III
Other Issues Related to PCATD Safety




assumes that one hour in the training device equals one hour in the
airplane. These experts question whether the performances of the PCATD
students in the Embry-Riddle Aeronautical University and University of
Illinois studies were due to the PCATD, training received in the course
curriculum or in the airplane itself during the check ride. They said that
the previously approved flight training devices had also not demonstrated
their effectiveness. These experts believed that both PCATDs and flight
training devices should be included under one set of standards and that a
device should not be approved for instrument training credit hours until it
shows that (1) one hour in the device is as effective for training as one
hour in the airplane and (2) the number of credit hours now permitted are
the appropriate amounts that should be approved for instrument training.
According to these and several academic flight training and safety experts,
PCATDs are superior to the older, grandfathered flight training devices that
are still approved for the full 15 to 20 hours of instrument flight training as
well as to meet instrument recency (currency) requirements.17 Figure III.7
shows an older flight training device that was grandfathered in for use as a
flight training device.




17
 These older model flight training devices may be mechanically driven desktop models that are not
computer-driven.



Page 38                 GAO/RCED-99-143 Personal Computer-Based Aviation Training Device
                                         Appendix III
                                         Other Issues Related to PCATD Safety




Figure III.7: Example of Grandfathered
Flight Training Device




                                         Note: This is an ATC 610 tabletop device.

                                         Source: Air Safety Foundation.




                                         Despite anecdotal evidence that there has been little difference in course
                                         completion hours between instrument students who are trained to
                                         proficiency in the airplane and those trained partly using flight training
                                         devices, we found no research showing that this level of instrument
                                         training effectiveness has been documented either for PCATDs or approved
                                         flight training devices. However, a 1971 study of beginning flight students
                                         found that an older training device, the Link GAT-1, transferred
                                         100 percent of 11 hours of training value to the student’s airplane
                                         performance, with a training effectiveness ratio of one-to-one; in other
                                         words, under certain conditions for beginning students, one hour in a
                                         flight training device can save one hour of time the student would



                                         Page 39                 GAO/RCED-99-143 Personal Computer-Based Aviation Training Device
Appendix III
Other Issues Related to PCATD Safety




otherwise need in the airplane to reach proficiency. In a 1973 study, also
on beginning flight students, airplane training time savings appeared to be
greater than one-to-one in the first few hours of training, where one hour
in the device saved more than one hour in the airplane and appeared to
decline in effectiveness to less than one-to-one as time in the flight training
device rose to 11 hours.18

According to one expert, the assumption of one-to-one substitution of
training device hours for airplane hours, though important, is secondary to
other concerns because few students complete their instrument
coursework in only the minimum required 35 to 40 hours but instead
average about 70 total hours to reach proficiency.




18
 H.K. Povenmire and S.N. Roscoe, “An Evaluation of Ground-Based Flight Trainers in Routine
Primary Flight Training.” Human Factors (1971), 13 (2), pp. 109-116; _________,” Incremental
Transfer effectiveness of a ground-based general aviation trainer.” Human Factors (1973). 15 (6), pp.
534-542.



Page 40                  GAO/RCED-99-143 Personal Computer-Based Aviation Training Device
Selected Bibliography


               Amdor, S.L., F.W. Isley, and Byron J. Pierce. “Computer-Based
               Instruction/Simulator Program for Fighter Lead-In Training: Feasibility
               Research.” July 1988. Air Force Human Resources Laboratory, Air Force
               Systems Command, Brooks Air Force Base, Texas, AFHRL-TP-87-64.

               Andrews, Dee H., Lynn A. Carroll, and Herbert H. Bell. “The Future of
               Selective Fidelity in Training Devices,” March 1996, Human Resources
               Directorate, Aircrew Training Research Division, Mesa, Arizona,
               AL/HR-TR-1995-0195.

               Barber, Sarah. “From PCs to Flight Simulators: Cost Effective Aircrew
               Training Devices for Aeronautical Decision Making and Crew Resource
               Management.” The Ohio State University, Proceedings of the Ninth
               International Symposium on Aviation Psychology. Columbus, Ohio, 1997

               Benton, Charles J., Paul Corriveau, Jefferson M. Koonce, and William C.
               Tirre. “Development of the Basic Flight Instruction Tutoring System
               (BFITS),” January 1992, Final Technical Paper for Period
               September 1989-December 1991, Air Forces Systems Command, Brooks
               Air Force Base, Texas, AL-TP-1991-0060.

               Beringer, Dennis B., and Howard C. Harris, Jr. “Assessment of Cockpit
               Navigation and Memory Aids Using PC-Based Flight Simulation,” Civil
               Aeromedical Institute, Federal Aviation Administration, Oklahoma City,
               Oklahoma. Proceedings of the Eighth International Symposium on
               Aviation Psychology, Columbus, Ohio, 1995.

               Beringer, Dennis B. “Use of Off-The-Shelf PC-Based Flight Simulators for
               Aviation Human Factors Research,” April 1996, Civil Aeromedical
               Institute, Federal Aviation Administration, U.S. Department of
               Transportation, Oklahoma City, Oklahoma, DOT/FAA/AM-96-15.

               Burki-Cohen, Judith, Nancy N. Soja, and Thomas Longridge. “Simulator
               Platform Motion- The Need Revisited,” The International Journal of
               Aviation Psychology 8 (3), 293-317.

               Caro, Paul W. “Some Current Problems in Simulator Design, Testing and
               Use,” March 1977, Professional Paper 2-77, Air Force Office of Scientific
               Research, Air Force Systems Command, Bolling Air Force Base,
               HumRRO-PP-2-77.




               Page 41           GAO/RCED-99-143 Personal Computer-Based Aviation Training Device
Selected Bibliography




Collins, William E., and Carolyn S. Dollar. “Fatal General Aviation
Accidents Involving Spatial Disorientation: 1976-1992,” August 1996, Civil
Aeromedical Institute, Federal Aviation Administration, U.S. Department
of Transportation, Oklahoma City, Oklahoma, DOT/FAA/AM-96-21

Cordell, Captain Tom. “Computer-Based Procedural Training,” United
Airlines, Proceedings of the Sixth International Symposium on Aviation
Psychology, Columbus, Ohio, 1991.

Dennis, Kerry A., and Don Harris. “Computer-Based Simulation as an
Adjunct to Ab Initio Flight Training,” The International Journal of Aviation
Psychology 8 (3) (1998) pp. 261-276,

Emanuel, Tom W., Jr., Henry L. Taylor, Charles L. Hulin, Gavan Lintern,
Sybil I. Phillips, and Donald A. Talleur. “Development of an Experimental
Flight Syllabus for Testing PC Computer-Based Aviation Training Devices,
”University of Illinois, Proceedings of the Eighth International Symposium
on Aviation Psychology, Columbus, Ohio, 1995, p. 1198

Emanuel, Tom W., Jr. “PC Computer-Based Aviation Training Devices,
Curriculum Development, and Student Affect,” University of Illinois,
Proceedings of the Ninth International Symposium on Aviation
Psychology, Columbus, Ohio, 1997, p. 1198.

Etem, Kamil, and Marcia Patten. “Communications-Related Incidents in
General Aviation Dual Flight Training,” Proceedings of the Ninth
International Symposium on Aviation Psychology, Columbus, Ohio, 1997,
p. 1204.

Giffin, Walter C., and Thomas H. Rockwell. “A Methodology for Research
on VFR Flight into IMC,” Proceedings of the Fourth International
Symposium on Aviation Psychology, Columbus, Ohio, 1987, p. 278.

Gopher, D., M. Weil, & T. Bureket. “Transfer of skill from a computer
game trainer to flight,” Human Factors 36 (3) (1994), pp. 387-405.

Hampton, Steven. “PC-Based Simulation: An Alternative for Teaching
Instrument Flying Skills,” Embry-Riddle Aeronautical University, Daytona
Beach, Florida, in PC-Based Instrument Flight Simulation – A First
Collection of Papers, A. Robert Sadlowe, editor. Presented at the Winter
Annual Meeting of the American Society of Mechanical Engineers,
December 1-6, 1991.



Page 42                 GAO/RCED-99-143 Personal Computer-Based Aviation Training Device
Selected Bibliography




William Moroney, Tom Kirton, and David W. Biers. “The Use of Personal
Computer-Based Training Devices in Teaching Instrument Flying: A
Comparative Study,” June 15, 1994, Prepared for the Federal Aviation
Administration, CAAR 15471931, Grant No. 92-G-0015 [Summarized in this
report.].

Koenig, Robert L. “Standard Computer Hardware and Software
Configured to Produce Useful Flight Simulator in Study.” Human Factors
& Aviation Medicine 43, No. 4 (July-August 1996), Flight Safety
Foundation.

Koonce, Jefferson M., “Predictive Validity of Flight Simulators as a
Function of Simulator Motion.” Human Factors 21 (2) (1979), pp. 215-223.

Koonce, Jefferson M., and William J. Bramble, Jr. “Personal
Computer-Based Flight Training Devices.” The International Journal of
Aviation Psychology 8 (3), (1998), pp. 277-292.

_____, Steven L. Moore, and Charles J. Benton. “Initial Validation of a
Basic Flight Instruction Tutoring System (BFITS),” Proceedings of the
Eighth International Symposium on Aviation Psychology, Columbus, Ohio,
1995.

Lintern, Gavan. “Transfer of Landing Skill after Training with
Supplementary Visual Cues,” Human Factors 22 (1), (1980), pp. 81-88.

_____, Roscoe, S.N., Koonce, J.M., and Segal, L.D. “Transfer of Landing
Skill in Beginning Flight Training.” Human Factors 32, (1990), pp. 319-327.

_____, Henry L. Taylor, Jefferson M. Koonce, and Donald Talleur. “An
Incremental Transfer Study of Scene Detail and Field of View Effects on
Beginning Flight Training,” Proceedings of the Eighth International
Symposium on Aviation Psychology, Columbus, Ohio, 1995, p. 737.

Lombardo, D.A. “Effectiveness of Computer-Based Flight Simulation,” in
B.M. Barker (Ed.), Collegiate Aviation Review, University Aviation Fall
Conference, Atlanta, 1993.

Moroney, William F., Steven Hampton, and David W. Biers.
“Considerations in the Design and Use of Personal Computer-Based
Aircraft Training Devices (PCATDs) for Instrument Flight Training: A Survey
of Instructors,” University of Dayton, Ohio, and Embry-Riddle



Page 43                 GAO/RCED-99-143 Personal Computer-Based Aviation Training Device
Selected Bibliography




Aeronautical University, Daytona Beach, Florida, Proceedings of the Ninth
International Symposium on Aviation Psychology, Columbus, Ohio, 1997.

Moroney, William F. and Brian W. Moroney, “Utilizing a Microcomputer
Based Flight Simulation in Teaching Human Factors in Aviation,”
Proceedings of the Human Factors Society 35th Annual Meeting, 1991, p.
523

“Nall Report: 1997.” Airplane Owners and Pilots’ Association Air Safety
Foundation, Frederick, Maryland.

O’Hare, David, and Stanley Roscoe. Flightdeck Performance: The Human
Factor. Ames, Iowa: Iowa State University Press; 1990.

O’Hare, David, and David Chalmers. “Risk Factors for Aviation Accidents
and Incidents: A Nationwide Study,” Proceedings of the Ninth
International Symposium on Aviation Psychology, Columbus, Ohio, 1997,
p. 1352.

Ortiz, Gustavo. “Effectiveness of PC-Based Flight Simulation.” The
International Journal of Aviation Psychology 4 (3), p. 285, 1994.

_____, Dominik Kopp, and Thomas Willenbucher. “Instrument Training
Using a Computer-Based Simulation Device at Luftansa Airlines,”
Andrews University, Berrien Springs, Michigan, and Luftansa Airlines,
Bremen, Germany, Proceedings of the Eighth International Symposium on
Aviation Psychology, Columbus, Ohio, 1995.

_____, Phillips, Sybil I., Charles L. Hulin, and Paul J. Lamermayer. “Uses of
Part-Task Trainers in Instrument Flight Training.” University of Illinois,
Proceedings of the Seventh International Symposium on Aviation
Psychology, Columbus, Ohio, 1993.

_____, Henry L. Taylor, Gavan Lintern, Charles L. Hulin, Tom W. Emanuel,
Jr.; Donald A. Talleur. “Developing Performance Measures for Evaluating
Personal Computer-Based Aviation Training Devices Within an FAR Part
141 Pilot Training School,” Proceedings of the Eighth International
Symposium on Aviation Psychology, Columbus, Ohio, 1995, p. 1204-1209.

Pierce, Rodney, and Dr. Richard S. Jensen. “PC Based Assessment and
Training of Aviation Decision Making Skills, The Ohio State University,




Page 44                 GAO/RCED-99-143 Personal Computer-Based Aviation Training Device
Selected Bibliography




Columbus, Ohio, Proceedings of the Ninth International Symposium on
Aviation Psychology, Columbus, Ohio, 1997.

Povenmire, H. K., & S. N. Roscoe. “An evaluation of ground based flight
trainers in routine primary flight training.” Human Factors 13 (1971),.pp.
109-116.

_____. “Incremental transfer effectiveness of a ground-based general
aviation trainer. ”1973. Human Factors 15, pp. 534-542.

Rogers, Brian K., Capt., U.S. Air Force. “Microcomputer-Based Instrument
Flight Simulation: Undergraduate Pilot Training Student Attitude
Assessment,” December 1991, Human Resources Directorate, Aircrew
Training Research Division, Williams Air Force Base, Arizona,
AL-TR-1991-0039.

Rolfe, Dr. J.M., J.R. Cook, and Squadron Leader C.G. Durose. “Knowing
What We Can Get from Training Devices: Substituting a Little Arithmetic
for a Measure of Emotion,” Proceedings of the Third International
Symposium on Aviation Psychology, Columbus, Ohio, 1985, p. 617.

Roscoe, Stanley N. “Simulator Qualification: Just as Phony as it Can Be,”
Illiana Aviation Sciences Limited, International Journal of Aviation
Psychology 1, No. 4 (1991), p. 335-339.

_____.( ed)., Aviation Psychology. Ames, Iowa: Iowa State University
Press; 1980.

Salas, Eduardo, Clint A. Bowers, and Lori Rhodenizer. “It is Not How
Much You Have but How You Use It: Toward a Rational Use of Simulation
to Support Aviation Training.” International Journal of Aviation
Psychology 8 (3) (1998), pp. 197-208

Svoboda, J.V., Ruth M. Heron, and H. Weinberg. “Emergency Manoevre
Pilot Training in a Low-Cost Flight Simulator,” Proceedings of the Seventh
International Symposium on Aviation Psychology, Columbus, Ohio, 1993,
p. 721.

Talleur, Donald, Henry L. Taylor, Gavan Lintern, Charles L. Hulin, Tom W.
Emanuel, Jr., and Sybil I. Phillips. “Effectiveness of an Integrated
Instrument Training Syllabus Using a Personal Computer and a Certified




Page 45                 GAO/RCED-99-143 Personal Computer-Based Aviation Training Device
           Selected Bibliography




           Ground Trainer,” University of Illinois, Proceedings of the Ninth
           International Symposium on Aviation Psychology, Columbus, Ohio, 1997.

           Taylor, Henry L., Gavan Lintern, Charles L. Hulin, Donald Talleur, Tom
           Emanuel, and Sybil Phillips; “Transfer of Training Effectiveness of
           Personal Computer-Based Aviation Training Devices,” November 1996,
           Final Technical Report ARL-96-3-/FAA-96-2, prepared for the Federal
           Aviation Administration, Oklahoma City, OK, Contract DTFA-94-G-044
           [Summarized in this report.].

           Wiener, Earl L. “Life in the Second Decade of the Glass Cockpit,” Keynote
           Address, Seventh International Symposium on Aviation Psychology,
           Proceedings of the Seventh International Symposium on Aviation
           Psychology, Columbus, Ohio, 1993, p. 1.

           Williams, Kevin W., and Robert E. Blanchard. “Development of
           Qualification Guidelines for Personal Computer-Based Aviation Training
           Devices,” February 1995, Final Report, Civil Aeromedical Institute,
           Federal Aviation Administration, U.S. Department of Transportation,
           Oklahoma City, Oklahoma, DOT/FAA/AM-95-6.

           Williams, Kevin W.,“Qualification Guidelines for Personal
           Computer-Based Aviation Training Devices: Instrument Training,”
           February 1996, Final Report, Civil Aeromedical Institute, Federal Aviation
           Administration, U.S. Department of Transportation, Oklahoma City,
           Oklahoma, DOT/FAA/AM-96-8.

           _____ (ed.) “DRAFT Proceedings of the Joint Industry-FAA Conference on
           Development and Use of PC-Based Aviation Training Devices: June 16-17,
           1994, Oklahoma City, OK,” July 1994, Civil Aeromedical Institute, Federal
           Aviation Administration, U.S. Department of Transportation, Oklahoma
           City, Oklahoma.

           Williams, Donna M., Robert W. Holt, and Deborah A. Boehm-Davis.
           “Training for Inter-Rater Reliability: Baselines and Benchmarks,”
           Proceedings of the Ninth International Symposium on Aviation
           Psychology, Columbus, Ohio, 1997, pp. 514-520.

           Wilt, Donna Forsyth. “Clueless in the Cockpit: Situation Awareness of
           Instrument Flight Students,” Proceedings of the Ninth International
           Symposium on Aviation Psychology, Columbus, Ohio, 1997, p. 1464.




(348100)   Page 46                 GAO/RCED-99-143 Personal Computer-Based Aviation Training Device
Ordering Information

The first copy of each GAO report and testimony is free.
Additional copies are $2 each. Orders should be sent to the
following address, accompanied by a check or money order
made out to the Superintendent of Documents, when
necessary. VISA and MasterCard credit cards are accepted, also.
Orders for 100 or more copies to be mailed to a single address
are discounted 25 percent.

Orders by mail:

U.S. General Accounting Office
P.O. Box 37050
Washington, DC 20013

or visit:

Room 1100
700 4th St. NW (corner of 4th and G Sts. NW)
U.S. General Accounting Office
Washington, DC

Orders may also be placed by calling (202) 512-6000
or by using fax number (202) 512-6061, or TDD (202) 512-2537.

Each day, GAO issues a list of newly available reports and
testimony. To receive facsimile copies of the daily list or any
list from the past 30 days, please call (202) 512-6000 using a
touchtone phone. A recorded menu will provide information on
how to obtain these lists.

For information on how to access GAO reports on the INTERNET,
send an e-mail message with "info" in the body to:

info@www.gao.gov

or visit GAO’s World Wide Web Home Page at:

http://www.gao.gov




PRINTED ON    RECYCLED PAPER
United States                       Bulk Rate
General Accounting Office      Postage & Fees Paid
Washington, D.C. 20548-0001           GAO
                                 Permit No. G100
Official Business
Penalty for Private Use $300

Address Correction Requested