United States General Accounting Office GAO Report to the Subcommittee on Aviation, Committee on Transportation and Infrastructure, House of Representatives June 1999 AVIATION SAFETY FAA’s New Inspection System Offers Promise, but Problems Need to Be Addressed GAO/RCED-99-183 United States GAO General Accounting Office Washington, D.C. 20548 Resources, Community, and Economic Development Division B-281502 June 28, 1999 The Honorable John J. Duncan Chairman The Honorable William O. Lipinski Ranking Democratic Member Subcommittee on Aviation Committee on Transportation and Infrastructure House of Representatives The aviation industry has forecast a potential 66-percent increase in passenger travel from 1999 to 2008. The U.S. aviation accident rate, which has remained relatively constant over the past two decades,1 must be substantially lowered to avoid escalating numbers of aviation deaths as air traffic increases. A key to reducing the aviation accident rate is for the Federal Aviation Administration (FAA) to have an effective process for inspecting the nation’s airline operations. In the past, we and others have expressed concerns about the adequacy of FAA’s inspection process to meet that challenge. Concerns about the inspection process focused on unstructured, nonsystematic inspections that produced few reports of safety problems and on the adequacy of inspectors’ technical training. These concerns also raised questions about the quality and consistency of the resulting inspection data and their usefulness for conducting analyses and targeting FAA’s resources to the greatest safety risks. FAA has responded to these concerns by redesigning the safety inspection system that it uses to oversee the nation’s airlines. FAA began using the revised approach, called the Air Transportation Oversight System (ATOS), for a limited number of airlines during the system’s initial implementation on October 1, 1998. Currently, the nation’s 10 largest passenger airlines are under ATOS.2 At your request, we reviewed FAA’s implementation of the new system. This report summarizes our work by addressing the following questions: • To what extent does ATOS address past concerns about FAA’s aviation safety inspections? • What factors, if any, surfaced during the implementation of ATOS that could impede its success? 1 The National Transportation Safety Board’s statistics show an accident rate of 5 fatal accidents for each 10 million flights on scheduled and nonscheduled service by U.S. airlines operating under part 121 of the Federal Aviation Regulations from 1982 through 1998. 2 These airlines are Alaska, America West, American, Continental, Delta, Northwest, Southwest, Trans World, United, and US Airways. Page 1 GAO/RCED-99-183 FAA’s New Aviation Inspection System B-281502 • What is FAA doing to address any factors that could impede the success of ATOS? The Air Transportation Oversight System is largely responsive to past Results in Brief concerns raised about key aspects of FAA’s aviation safety inspections and the usefulness of inspection data. These concerns centered on FAA’s unstructured inspection process, the adequacy of technical training for inspectors, the quality and consistency of inspection data, and the usefulness of those data for identifying safety problems and targeting the agency’s resources to the greatest risks. Addressing these concerns involved a fundamental redesign of the way FAA inspects the nation’s airlines. To improve inspection quality, the new program emphasizes a system safety approach that goes beyond spot-checking airlines for compliance with Federal Aviation Regulations. Using safety principles originally created for the nuclear industry, it calls for a systematic review of airlines’ policies and procedures to ensure that they incorporate basic safety principles, such as clear lines of responsibility and written documentation. It fosters more consistent, structured inspections by standardizing inspection tasks, linking inspectors’ training more closely to their assigned responsibilities, and using teams rather than individual inspectors to perform many inspections. The program also calls for a number of enhancements to improve the usefulness of inspection data for analysis and targeting. They include a standardized database for reporting inspection results and the addition of data quality assurance managers and analysts. The goal of this redesign is to target inspection resources to those areas that present the greatest safety risks. ATOS offers promise for significantly strengthening FAA’s inspection process, but FAA must also address the problems identified in this report to ensure that the new system fulfills its promise. FAA’s ability to conduct effective inspections remains limited by a lack of clear guidance, staff turnover, and continued difficulties with the adequacy of inspectors’ technical training and experience. The anticipated enhancements to make inspection data more useful have not been achieved because of problems with reporting requirements and the incompatibility of the program’s database with FAA’s primary inspection analysis system. In addition, FAA planned to hire an analyst for each of its new inspection teams to analyze inspection data for safety trends and to guide inspection planning, but has not yet done so because of higher priorities, such as increasing salaries for air traffic controllers. These problems resulted largely from FAA’s decision to implement the new inspection system on an overly ambitious schedule. Page 2 GAO/RCED-99-183 FAA’s New Aviation Inspection System B-281502 Meeting FAA’s target date for implementation meant that complex, critical steps—such as developing guidance, training inspectors, creating databases, and consulting with the affected airlines—had to be compressed into a very short time. FAA has begun to address some of the problems with the program. FAA management officials acknowledged that ATOS faces significant challenges. They agreed with our conclusion that the program should not be expanded beyond the nation’s 10 major airlines until the problems that emerged during the program’s initial implementation are resolved. However, some of these problems have not yet been fully addressed. Consequently, we recommend several specific actions to clarify the program guidance and improve the usefulness of FAA’s database for targeting inspection resources to the areas of greatest potential safety risk. Federal law establishes that the safety of U.S. air passengers is a joint Background responsibility of the airlines and FAA. The airlines are responsible for operating their aircraft safely. FAA is responsible for, among other things, examining an airline’s operations when the airline seeks a certificate to operate and for conducting periodic inspections to ensure continued compliance with safety regulations. Within FAA, the Office of Flight Standards Service develops the Federal Aviation Regulations that airlines must follow and prepares guidance on how FAA’s safety inspectors should perform inspections. FAA has nearly 3,300 safety inspectors located in 101 district offices throughout the United States. One of the inspectors’ primary functions is conducting what FAA calls “routine surveillance”—a process of continuous periodic safety inspections of airlines and aviation-related activities.3 These inspections include having an inspector visually spot-check an airplane at the gate, monitor procedures on a scheduled flight, or observe maintenance being performed on an aircraft or its component parts. The inspections cover four main areas: • Operations inspections focus on such items as pilots’ certification and performance, flight crews’ training, and in-flight record keeping. • Maintenance inspections examine an airline’s overall maintenance program, including the training of aviation mechanics, the development of 3 Other primary functions include certifying airlines’ operations, investigating accidents and incidents, and taking other steps to promote safety. Page 3 GAO/RCED-99-183 FAA’s New Aviation Inspection System B-281502 maintenance manuals, and procedures for repairing aircraft and their components. • Avionics inspections focus on electronic components of the aircraft. • Cabin safety inspections concentrate on cabin procedures, passenger safety, and carry-on baggage. In recent years, we and others have reported on problems with FAA’s inspections.4 Reacting to these reports and to a series of fatal aviation accidents in the mid-1990s, FAA conducted a number of studies of its own, including the report entitled FAA 90 Day Safety Review, issued in September 1996.5 This review recommended that inspections be made more systematic and that they be targeted to deal with identified risks, such as airlines’ financial instability and significant contracting out of maintenance. ATOS, developed by the Office of Flight Standards Service and adapted in part from safety principles originally created for the nuclear industry, resulted from these recommendations. FAA incorporated features in ATOS to address past concerns about The ATOS Concept Is inspection quality and the usefulness of inspection data for identifying Responsive to Many potential safety threats and for targeting resources to areas that pose the Past Concerns About greatest risk. Before developing its new aviation safety inspection system, FAA analyzed past concerns about its inspections. The resulting ATOS FAA’s Safety inspection concept focuses on ensuring that an airline has operating Inspections systems in place to control the potential hazards and risks of flying and to prevent accidents. FAA structured ATOS to evaluate both an airline’s operating systems and its adherence to those systems in day-to-day operations. In addition, FAA incorporated specific features into ATOS to make inspections more consistent, structured, and thorough and to improve the collection and analysis of inspection data. ATOS Focuses on System FAA emphasizes a system safety approach in ATOS that replaces routine Safety and Accident surveillance and goes beyond spot-checking airlines for compliance with Prevention aviation regulations. System safety involves the application of technical and managerial skills to identify, analyze, assess, and control hazards and risks. It covers every aspect of an airline’s operations, from the design of 4 For a list of relevant reports, see the bibliography and the section citing related GAO products at the end of this report. 5 Challenge 2000: Recommendations for Future Aviation Safety Regulation, prepared for the Federal Aviation Administration by Booz-Allen & Hamilton Inc. (Apr. 1996); and FAA 90 Day Safety Review, Federal Aviation Administration (Sept. 16, 1996). Page 4 GAO/RCED-99-183 FAA’s New Aviation Inspection System B-281502 the hardware to the culture and attitudes of the airline’s personnel. ATOS calls for a systematic review of an airline’s policies and procedures to ensure that they incorporate such basic safety principles as clear lines of responsibility and written documentation. FAA intended ATOS to ensure that an airline has and follows policies and procedures that build in system safety. FAA implemented ATOS on October 1, 1998, to cover the nation’s 10 major passenger airlines. Under ATOS, FAA assigns a team of inspectors to oversee each airline. Three principal inspectors lead the team, one for each major area of inspections (operations, maintenance, and avionics). Additional team members can be based in one of two ways. Inspectors based at the FAA office that holds the airline’s operating certificate work full time on the ATOS team. In contrast, field inspectors, who work in other locations to which the airline flies, work part time on the ATOS team and complete additional duties, such as accident investigations and other inspections, for the local FAA office to which they are assigned. About 540 inspectors are assigned to the 10 ATOS teams. Each ATOS team also includes one cabin safety specialist, whose inspections focus on such areas as flight attendants’ training, carry-on baggage, and emergency evacuation procedures. FAA included two kinds of guidance in ATOS to help a team plan and carry out inspections of the airline it oversees. First, automated ATOS planning guidance is used to develop the comprehensive surveillance plan for each airline. The planning guidance calls for using existing safety data, risk indicators, and the inspectors’ knowledge of an airline’s operations to determine the priority and frequency of inspection activities. The resulting comprehensive surveillance plan includes a series of inspection tasks to determine whether the airline has systems in place to ensure safety and a second series of inspections to verify that the airline is actually using those systems. FAA also developed ATOS guidance for conducting inspections that is intended to describe the tasks to be performed for each type of inspection. For a more detailed description of the ATOS guidance and the development of the comprehensive surveillance plan, see appendix I. FAA designed ATOS to be improved on an ongoing basis. FAA has established an ATOS Program Office to formulate and implement changes to ATOS and to support FAA inspection teams through a hotline, help desk, and Web site. FAA has also established an internal audit team of aviation safety inspectors to evaluate the program, the System Process Audit Group. This internal audit team is an independent FAA organization that reports directly to the Director of Flight Standards. Page 5 GAO/RCED-99-183 FAA’s New Aviation Inspection System B-281502 ATOS Addresses Many In the past, Flight Standards typically allocated a large portion of its Past Concerns About inspection resources to thousands of unstructured inspections that Inspection Quality produced few reports of problems. We reported last year, for example, that inspectors reported no problems for 96 percent of the inspections they conducted in fiscal years 1990 through 1996.6 Reviews of FAA’s inspection program suggest that FAA detects more problems through rigorous structured inspections than through unstructured inspections. Our recent review of FAA’s oversight of the facilities airlines use to repair aircraft confirmed that standardizing inspection tasks through the use of checklists promotes more comprehensive inspections.7 Past concerns also included problems with inspectors’ training. Specifically, inspectors have performed inspections for which they did not have appropriate or current credentials, in part because of limited funding for training. Providing adequate technical training for FAA’s inspector workforce has proven difficult because of the rapid change in aviation technology. In addition, airlines can meet regulatory requirements in a variety of ways, making it difficult for FAA’s inspectors who inspect many different airlines to be familiar with the FAA-approved procedures of each airline. Both principal inspectors and airline officials we interviewed said that this lack of familiarity sometimes resulted in airlines’ being unfairly cited for noncompliance. Finally, our review of aircraft repair facilities noted that individual inspectors generally identify far fewer deficiencies than teams do. The unstructured inspection activities and the underreporting of violations by inspectors resulted in inaccurate, incomplete, and inconsistent information that was not very useful for analyzing safety risks or targeting the agency’s resources to the problems that pose the greatest risks. FAA included features in ATOS to move toward more consistent, structured inspections by using a system safety approach and by providing new, standardized inspection tasks. FAA developed automated ATOS planning guidance to ensure that inspectors use the same criteria to determine the annual inspection activities for each of the major airlines. An ATOS team uses the planning guidance to identify potential problem areas at each airline that should be inspected more frequently. Similarly, to standardize inspection activities across airlines, the guidance for conducting inspections lists tasks for each inspection. The inspection guidance is designed to ensure that each inspector looks at an airline’s systems and 6 Aviation Safety: Weaknesses in Inspection and Enforcement Limit FAA in Identifying and Responding to Risks (GAO/RCED-98-6, Feb. 27, 1998). 7 Aviation Safety: FAA Oversight of Repair Stations Needs Improvement (GAO/RCED-98-21, Oct. 24, 1997). Page 6 GAO/RCED-99-183 FAA’s New Aviation Inspection System B-281502 elements in the same way. It is also supposed to serve as a checklist that inspectors can use to ensure that their inspections are thorough. While the old inspection system listed multiple tasks that could be performed, inspectors were not required to complete any specific tasks or to document which tasks they performed. Thus, the scope of the inspection work actually completed could not be determined. In contrast, ATOS requires inspectors to document whether the airline being inspected complied with each item on the inspection checklist. The checklist also serves as a template for reporting inspection results in the ATOS database. FAA also incorporated team inspections in the ATOS approach. We have reported in the past that teams have been more effective than individuals in identifying areas where airlines were not in compliance with FAA regulations. In many cases, the deficiencies identified by teams are systemic and long-standing. Under ATOS, teams will identify deficiencies and plan inspections. Many inspections will be performed by teams rather than by individual inspectors, as has been done in the past. Individual inspectors will continue to perform some of the inspection work identified in the plan. FAA also included several features in ATOS to address past concerns about inspectors’ training by more effectively linking inspectors’ technical training and qualifications to their job responsibilities. First, FAA designed ATOS to link inspection assignments to the technical background of each inspector and to identify any additional technical training needed to accomplish the work plan. Inspectors cannot adequately inspect aircraft or systems unless they have had the appropriate technical training. Second, inspectors assigned to an airline, including field inspectors, must complete training on both ATOS and the airline’s specific policies and procedures before they can conduct inspection activities. The training on ATOS provides an overview of the system safety concept and how it differs from FAA’s past inspection approach. The training on the airline’s policies and procedures familiarizes inspectors with the approved operating procedures of the airline they oversee. Page 7 GAO/RCED-99-183 FAA’s New Aviation Inspection System B-281502 ATOS Addresses Many FAA needs complete, accurate inspection data to target its limited Past Problems With the inspection resources to the areas that pose the greatest potential safety Usefulness of Inspection risks. We reported in 1989 and again in 1991 on inaccurate and incomplete data in FAA’s inspection database and recommended in 1995 that FAA Data develop a comprehensive and coordinated strategy for improving its data.8 More recently, we reported last year that 70 percent of Flight Standards’ inspectors did not enter all of the violations they found into their inspection tracking system in fiscal year 1996.9 Some inspectors said they did not report violations when compliance could be achieved informally by bringing problems to the attention of the airlines. Others said they handled less serious violations informally because the paperwork involved in reporting violations was too burdensome. FAA is implementing a streamlined procedure for documenting and processing minor administrative violations, which should better enable the agency to target its limited inspection resources to the areas that pose the greatest risks. The streamlined procedure will reduce paperwork for some types of enforcement cases, but other efforts will be needed to ensure the complete, accurate inspection data needed for improved targeting of inspection resources. FAA included several features in ATOS to address past concerns about the usefulness of inspection data for analysis and targeting. First, the standardization of inspections and the development of guidance for planning and conducting inspections are steps intended to improve the quality of FAA’s data by making inspections more systematic and thorough. When inspections are more standardized across airlines, data quality is improved. Second, FAA created a new position within the ATOS team overseeing each airline: a data evaluation program manager, whose job will be to review data for validity, accuracy, and completeness before they are finalized in the ATOS database for analysis. ATOS also added a new position for an analyst on each team. The analyst is responsible for collecting and analyzing data to support inspection planning and retargeting. Finally, FAA included features in ATOS to improve the targeting of inspection resources. FAA designed ATOS to allow the targeting of inspections based on an airline’s size, operations, past history, and known problem areas. The automated planning guidance can be used to indicate the risk factors applicable to the airline, such as whether an airline is a 8 Aviation Safety: FAA’s Inspection Management System Lacks Adequate Oversight (GAO/RCED-90-36, Nov. 13, 1989); Aviation Safety: Problems Persist in FAA’s Inspection Program (GAO/RCED-92-14, Nov. 20, 1991); and Aviation Safety: Data Problems Threaten FAA Strides on Safety Analysis System (GAO/AIMD-95-27, Feb. 8, 1995). 9 GAO/RCED-98-6, Feb. 27, 1998. Page 8 GAO/RCED-99-183 FAA’s New Aviation Inspection System B-281502 new entrant or contracts with other companies for its maintenance, training, or ground handling services. Based on the risk factors, the comprehensive surveillance plan targets particular areas. FAA also recognized that needed inspection work must drive the assigned inspection resources for each airline. Finally, ATOS gives inspectors the flexibility to retarget resources at any point during the year based on inspection results. This flexibility allows FAA to focus on new problems as they surface, rather than waiting until the next year’s work. Because ATOS currently focuses on major airlines, it does not address concerns about the need to provide additional oversight of new entrant airlines (that is, airlines in their first 5 years of operation). A separate FAA initiative, the Certification Standardization and Evaluation Team, has standardized and automated the process for granting operating certificates to new airlines. ATOS system safety concepts have been integrated into the certification process for new airlines. The new certification concept includes a national team to assist local district offices in reviewing the applications of new airlines and monitoring these airlines for their first 5 years of operation. As new airlines receive certification, FAA plans to oversee them using the ATOS program. FAA is not, however, currently providing any additional oversight of new entrant airlines that were already in operation prior to the new certification process. Problems that emerged during design and implementation limit the ATOS’ Design and potential of the ATOS concept to bring about needed improvements in FAA’s Implementation aviation safety inspections. Problems with the ATOS inspection guidance, Problems Limit FAA’s the links between inspectors’ qualifications and their work assignments, and assembling effective teams affect the improvements envisioned for Efforts to Improve inspection quality and the usefulness of inspection data for analysis and Safety Inspections targeting. In addition, FAA did not take advantage of the expertise of airline or industry representatives in developing ATOS. Design and Although ATOS calls for (1) more systematic, structured inspections, Implementation Problems (2) closer links between inspectors’ training and their assigned work Limit Improvements to responsibilities, and (3) greater use of team inspections to improve inspection quality, its success in the first 6 months has been limited: Inspection Quality • Inspection guidance is not complete and is not sufficiently clear and detailed to accomplish the systematic, structured inspections promised by the ATOS concept. Page 9 GAO/RCED-99-183 FAA’s New Aviation Inspection System B-281502 • ATOS has not resolved the long-standing problems with matching inspectors’ qualifications to their work assignments. • Team inspections are hampered by problems with assigning inspectors, including turnover, work locations that do not match inspection needs, an unwillingness of inspectors to travel, and FAA’s inability to estimate the resources needed to complete ATOS inspections. ATOS Inspection Guidance The new inspection guidance is not adequate to ensure the Needs Significant Improvement comprehensive, standardized inspections envisioned by the ATOS concept. We found several problems with the ATOS inspection guidance as implemented. Guidance does not cover all applicable regulations. One basic purpose of inspections is to ensure that an airline complies with Federal Aviation Regulations. Although ATOS aims to go beyond ensuring compliance to see whether safety is built into an airline’s operations, compliance should also be confirmed. Principal inspectors we interviewed expressed several concerns about the link between the ATOS inspection guidance and aviation regulations. One group of inspectors analyzed the ATOS inspection guidance to determine whether it covered all applicable Federal Aviation Regulations. They identified 296 specific regulatory requirements that the ATOS inspection guidance did not address out of approximately 2,300 applicable requirements. While we have not verified each of the 296 requirements said to be missing, our sampling of the inspectors’ results confirmed that ATOS overlooked some key regulatory requirements. For example, the inspection guidance does not cover regulations requiring airline employees to be trained to handle hazardous materials. Because the ATOS guidance does not include all applicable regulatory requirements, inspections may not be thorough enough to ensure compliance. Some guidance is not applicable to ATOS airlines. In addition to the regulatory requirements not covered by ATOS, principal inspectors we interviewed identified a number of regulations referenced in ATOS that are not applicable to the major airlines currently under ATOS. For example, FAA requires that major airlines maintain an aircraft’s weight and balance to ensure that it remains within approved limits. However, the ATOS inspection guidance for the weight and balance program is based on the FAA regulations governing commuter airlines.10 Because ATOS currently covers only the largest airlines, the inspection guidance should exclude 10 Commuter airlines are those that conduct scheduled passenger-carrying operations in aircraft that have 10 or fewer seats and operate under part 135 of the Federal Aviation Regulations. Page 10 GAO/RCED-99-183 FAA’s New Aviation Inspection System B-281502 those regulations that are applicable to commuter airlines or other types of operators, such as general aviation aircraft. Principal inspectors we interviewed also questioned the appropriateness of some parts of the ATOS guidance that have no basis in regulatory requirements. For example, the ATOS guidance calls for reviewing the résumés of some airline officials, although regulations do not specify qualifications and experience for their positions. Principal inspectors also questioned the basis for some of the ATOS determinations that rely on very subjective judgments, such as whether the airline has a “safety focus.” They noted that the ATOS guidance does not distinguish inspection tasks and findings based on regulations, which are legally enforceable, from those based on such other sources as inspector handbook guidance, which is advisory. Safety officials at most of the major airlines echoed the inspectors’ concerns. Guidance is not sufficiently clear and detailed. The fact that ATOS provides guidance to inspectors on how to plan and perform their inspections represents a major step toward the standardization of inspection tasks. However, the guidance it offers is not yet thorough or detailed enough to achieve that goal. Principal inspectors we interviewed questioned its usefulness, saying that it was not clear or detailed enough. They reported that they found the language of the guidance for planning inspections difficult to use because it does not detail the tasks to be performed well enough. Staff at Sandia National Laboratories, who were asked by FAA to comment on the ATOS program because of their expertise in system safety in the nuclear industry, had pointed to similar concerns before ATOS was implemented. In a report on ATOS, the Sandia staff noted that the inspection guidance was not based on analyses of specific ATOS inspection tasks and the recording of the results.11 The Sandia report noted that a proper task analysis describes the steps to take and the standards for determining that the results are correct and complete. The inspection guidance does not provide this level of detail. FAA’s guidance material was not thoroughly tested. The lack of clarity and detail in the ATOS guidance, both for (1) planning and (2) conducting inspections, reflects the fact that FAA did not thoroughly test and validate it before implementation. When FAA first tested the planning guidance in April 1998, the inspectors involved said they had trouble using it because the questions were too vague and broad. When FAA later tested a revised 11 “Analysis and Data Issues for the Implementation of a Systems Safety Focus in Air Transportation Oversight,” Oct. 6, 1998. Page 11 GAO/RCED-99-183 FAA’s New Aviation Inspection System B-281502 automated version of the planning guidance, most test participants were members of the ATOS work group. Work group members filled in the answers without actually performing the relevant analysis or inspection tasks. While FAA tested the planning guidance, it never tested the guidance for conducting inspection tasks. Flight standards did not allow sufficient time for ATOS’ implementation. While FAA spent several years developing the ATOS concept, the target implementation date left only 7 months to take the agency’s new inspection program from concept to reality. The target date of October 1, 1998, did not allow time for the adequate development, testing, and validation of new inspection procedures and tracking systems or for training and preparing the inspector workforce. The ATOS work group established in February 1998 faced multiple challenges in meeting the implementation deadline in the allotted 7 months. These challenges included (1) implementing a completely reengineered process for conducting inspections, (2) developing new automated systems for planning ATOS inspections and tracking their results, and (3) preparing FAA for the cultural changes involved in having inspectors perform their work differently. FAA did not adequately train inspectors to use ATOS. The full impact of the shortcuts taken to meet the implementation deadline became apparent during the inspectors’ training and, subsequently, as the teams of inspectors used the new ATOS planning guidance to develop a comprehensive surveillance plan for each major airline and to initiate inspections. FAA trained more than 800 inspectors, managers, and other FAA staff on the ATOS concept and planning guidance from September through November 1998. Because of the speed with which the ATOS guidance and tracking systems were developed, FAA had little time to develop training for its inspectors. For example, very few of the 88 ATOS lists of inspection tasks were available in time for the training sessions, and the database for reporting inspection results was not yet completed. Participants in the training sessions repeatedly noted that the implementation of ATOS had been rushed and that there were too many unknowns to allow for this initiative to go smoothly. In course evaluations and in our interviews, inspectors said that they were not adequately trained on (1) how to perform the new inspection tasks, (2) how to record the results of their inspections in the ATOS database, and (3) how to use the data on inspections to retarget resources. The ATOS internal audit team concluded, “The ATOS training . . . Page 12 GAO/RCED-99-183 FAA’s New Aviation Inspection System B-281502 dedicated an insufficient amount of time to training on the [inspection guidance].” Standardized inspections and reporting will not occur without clear, understandable guidance, detailed information on tasks to be completed, and training in both conducting inspections and reporting findings. Links Between Inspectors’ The ATOS model for linking inspectors’ qualifications and training to Qualifications and Assignments assignments is not yet a reality. We found that some inspectors still lack Are Not Fully Established experience with FAA or major airlines or lack the specific training needed to perform their jobs effectively. Lack of experience with FAA or major airlines. In selecting staff to fill positions on ATOS teams, FAA did not effectively match the qualifications of field inspectors to their new positions. The selection of field inspectors to fill ATOS assignments was not based on any nationwide criteria or guidance. Principal inspectors told us that they had no say in the selection of the field inspectors assigned to their teams. As a result, the qualifications of some field inspectors did not match the teams’ needs. Several principal inspectors and managers said that field inspectors assigned to them included newly hired staff who were unfamiliar with FAA or inspectors—including experienced FAA staff—who lacked background with major airlines. For example, one manager said that some of the field inspectors assigned to his team had no experience in large aircraft and lacked appropriate qualifications. FAA principal inspectors have told us that it takes several years to develop familiarity with the agency’s regulations and procedures or with an airline’s procedures. Consequently, newly hired personnel may need several years of experience before they can work independently and be fully productive. Similarly, inspectors trained on small aircraft may need extensive on-the-job training to understand the workings of a major airline. Principal inspectors questioned whether some team members had the appropriate skills to oversee a major airline. Lack of specific training. Even inspectors who have experience with major airlines may not have the specific technical qualifications to perform the ATOS inspection tasks planned for the airline to which they are assigned. Principal inspectors we interviewed said that the technical qualifications of a number of the inspectors assigned to them did not match those needed to oversee the airline in question. For example, one operations inspector said that three of his field inspectors did not have the appropriate license required to fly the aircraft used by the airline or to conduct flight checks to observe its pilots. Similarly, a principal avionics Page 13 GAO/RCED-99-183 FAA’s New Aviation Inspection System B-281502 inspector said that neither he nor his assistant had the appropriate technical training in the Airbus aircraft that the airline flies. Furthermore, the rapid changes in the aviation industry make it difficult to provide sufficient technical training to keep experienced inspectors current. These difficulties in keeping pace with technological advances reflect FAA’s persistent problems with providing technical training for inspectors and matching inspectors’ qualifications to their job assignments. Assignment Issues Affect the Staff turnover, travel requirements, and conflicting demands on inspectors’ Quality of Team Inspections time undermine FAA’s ability to assemble effective ATOS teams. Many of these assignment issues cannot be resolved within the context of the ATOS program alone because they involve broader decisions about inspectors’ responsibilities and Flight Standards’ staffing. Staff turnover. The turnover among field inspectors assigned to the airlines covered by ATOS has made it difficult to plan inspection work or to meet the ATOS training requirements. Principal inspectors said that they had lost field inspectors originally assigned to their teams, primarily because field inspectors went elsewhere in FAA to accept promotions that are not available to them under Flight Standards’ current staffing structure. For example, of the 28 inspectors assigned to one ATOS team, 11 had been reassigned, leaving only 17 available. Because some replacements lacked ATOS training, training on the airline’s policies and procedures, or both, they were ineligible to conduct inspections under ATOS. Principal inspectors were concerned about how, if turnover continues, to meet the requirement that inspectors be trained on the policies and procedures of the airline they oversee. In many cases, the airlines provided this training in cooperation with FAA. Several principal inspectors said that repeated requests for training would be burdensome to the airlines. Principal inspectors noted that the turnover of field inspectors would probably continue unless changes are made in Flight Standards’ grade and pay levels to permit field inspectors to be promoted. Inspectors’ work locations do not always match inspection needs. Because the work locations of some inspectors assigned to ATOS teams do not match inspection needs, the inspectors will have to travel to complete their work. This has made it difficult to assemble effective teams for several reasons. First, many of the principal inspectors we interviewed told us that field inspectors were assigned to locations where they are not needed, while parts of the country where the airlines have substantial activity have no field presence. For example, one principal inspector told us that he had a field inspector assigned to his team who was located in Page 14 GAO/RCED-99-183 FAA’s New Aviation Inspection System B-281502 Boise, Idaho, where the airline he oversees does not fly, but that he had no one in Kansas City, where he needs an inspector to oversee maintenance operations. Because of these problems, a number of principal inspectors suggested that they be given a role in identifying the needed qualifications and work locations for field inspectors assigned to the ATOS teams. Furthermore, in the past, teams overseeing an airline drew on field inspectors in FAA’s international offices to inspect foreign maintenance facilities and other overseas operations of airlines with international routes. Because FAA has not trained inspectors in its international offices to perform ATOS inspections, ATOS team members will have to travel overseas to complete planned inspections, despite FAA’s already limited travel funds. ATOS does not resolve resource constraints. FAA designed ATOS to address the inspection portion of its inspectors’ workload and did not address the inspectors’ other responsibilities. In addition to inspections, the oversight of an airline includes ongoing activities referred to as “demand” work. Demand work includes certification and approvals for initiatives taken by the airline, such as adding new aircraft types to the fleet, adding new destinations, implementing computerized record keeping, and restructuring by management. The ATOS internal audit staff confirmed that a conflict exists between demand work and ATOS inspection work. Inspectors based near the offices of the airlines they oversee are especially prone to this conflict because they perform both demand work and ATOS inspections. Many principal inspectors said that these inspectors do not have sufficient time to complete both their demand work and ATOS inspection work. Field inspectors, who are assigned only part time to ATOS, are subject to different pressures. ATOS field inspectors must also investigate accidents and perform other inspection work for the local offices to which they are assigned. Principal inspectors expressed concern that field inspectors might not be available for ATOS work when needed because of demands from their local offices. In addition, several principal inspectors and managers questioned whether the emphasis on ATOS has shifted resources away from other areas of concern, such as repair stations, troubled smaller carriers, and general aviation safety. Managers contended that such conflicts between demand work and inspection activities will persist as long as ATOS focuses solely on inspections, rather than having a broader view of the work that inspectors perform. These conflicting demands on inspectors’ time and on resources such as travel funds are exacerbated because managers do not know what resources will be needed to perform ATOS inspections. The ATOS guidance Page 15 GAO/RCED-99-183 FAA’s New Aviation Inspection System B-281502 does not include estimates of how long various inspection tasks should take. Without such information, it is difficult to estimate how many inspectors will be needed to perform the tasks or what travel funds they may need to accomplish the planned work. Both inspectors and their managers told us that they are unable to estimate how much time many of the ATOS inspection tasks will require. Two principal inspectors told us that managers of some local offices will not approve ATOS work plans for field inspectors unless they know how much time will be needed to complete the work. Testing and validation of the ATOS guidance would have provided preliminary estimates for the time needed to complete various ATOS inspection tasks. The ATOS Data Collection Although ATOS includes initiatives to address past problems with the Process Limits the usefulness of FAA’s inspection data for analysis and targeting of resources Usefulness of Data for to the greatest safety risks, this goal may not be fully realized for several reasons. First, because of the problems with the ATOS guidance that we Analysis and Targeting have already discussed, the data collected from ATOS inspections are not likely to be reliable enough to support meaningful analyses. Furthermore, in translating ATOS from concept to design and eventual implementation, FAA did not adequately determine its data analysis needs. The volume of inspection data available for analysis has also dropped dramatically under ATOS because few inspections have been completed. Finally, principal inspectors responsible for overseeing airlines did not have timely access even to the limited data available until FAA granted them access in May 1999. FAA Did Not Adequately While ATOS calls for structured inspections intended to result in more Determine Data Analysis Needs thorough and consistent data, the way ATOS collects data limits the potential of its database as a tool for analysis and targeting. The data limitations reflect the fact that FAA did not sufficiently analyze its data needs before developing ATOS inspection guidance and its automated database. The development of an effective automated system begins with a thorough analysis of the data required to meet the needs of those using the database. In the case of ATOS, a thorough analysis would describe in detail the questions that need to be asked to improve aviation safety, determine precisely what data are needed to answer those questions, and plan the appropriate analyses to be conducted on those data to answer the questions. After the preliminary analysis of the users’ data needs is completed, the prototype system must be tested, validated, and revised in an iterative process between data users and automation developers. Staff from Sandia National Laboratories reviewed ATOS and concluded that its Page 16 GAO/RCED-99-183 FAA’s New Aviation Inspection System B-281502 developers did not go beyond abstract, high-level statements about users’ needs to the level of detail essential to ensure data quality. They added that without more detailed information it will be difficult to identify the data and information needed to answer questions about aviation safety and impossible to support the detailed automated analysis of an airline. Because FAA did not sufficiently analyze or list the data needed by inspectors to make determinations about aviation safety and did not adequately test or validate ATOS, features that would maximize the usefulness of data for analysis and targeting were not built into the system. We found four specific limitations with the ATOS database. Key information is not required. ATOS does not currently require inspectors to record inspection data that are essential for effective analyses. To perform effective analyses of safety data, basic information is needed, such as the airline’s name; the make, model, and series of the aircraft; the aircraft and pilot identification numbers; and where the inspection was performed. However, an ATOS inspection record can be closed without any of this basic information having been entered because the system requires only that inspectors indicate whether an airline is in compliance and explain any violations. For example, our review of completed ATOS inspections found 18 completed inspection activities related to de-icing operations for which the location had been entered, not in the appropriate field, but in a comment field. Data contained in comment fields cannot be used for automated analyses to determine, for example, which airports are experiencing problems with de-icing. In addition, the ATOS inspection guidance does not provide inspectors with an index or clear instructions on where in the ATOS database to report findings. If an inspector cannot easily figure out where to report a finding, it may go unreported. For example, an inspector who found incorrect safety placards on an aircraft said he had to search ATOS for nearly 4 hours to record this violation. Inspectors’ underreporting of violations that they observed has jeopardized the completeness and quality of inspection data in the past. ATOS does not resolve the problems that contributed to underreporting, and the difficulty in finding where to report violations in ATOS may exacerbate this problem. Response options preclude meaningful analyses. Many ATOS inspection activities are to be summed up in a report with a single “yes” or “no” response to the items on the inspection checklist for a given area. If an inspector finds problems during a single inspection activity, it results in a Page 17 GAO/RCED-99-183 FAA’s New Aviation Inspection System B-281502 “no” for the entire question. Regardless of whether an inspector finds one problem in 10 inspections or one in 100, a “no” results. Without knowing the proportion of “no” responses for a given item, FAA cannot use the inspection results to conduct quantitative analyses on ATOS data, for example, a trend analysis to determine whether the airline’s compliance in a given area has improved. Minimum number of inspection activities needed is not specified. ATOS currently does not specify the minimum number of activities needed to complete each inspection. Instead, individual inspectors decide independently how many times inspection observations should be performed to determine whether the airline follows its procedures and complies with regulations. This determination is subjective. In our review of the ATOS data available as of March 31, 1999, we found that four inspectors conducted anywhere from 2 to 12 observations to complete the record for the same type of inspection. Because these reports result in “yes” or “no” answers on the completed inspection report, rather than in a quantifiable report of the proportion of “no” answers, the data submitted by these inspectors cannot be compared or analyzed. Because inspectors will never have enough time to observe every safety-related component of an airline’s system, it is important to define the minimum number of inspections to be conducted and to report accurately the proportion of instances of noncompliance. ATOS does not link to Flight Standards’ existing data analysis system. The information in the ATOS database cannot be analyzed by Flight Standards’ existing aviation risk analysis system, the Safety Performance Analysis System (SPAS). FAA has spent $95 million developing SPAS to analyze key aviation safety data, identify trends and potential safety concerns, and target inspection resources accordingly. In January 1998, the ATOS development team noted that ATOS inspection data could not be analyzed by SPAS. The development team recommended that ATOS not be implemented until FAA could develop appropriate links between ATOS and SPAS. However, FAA, in its efforts to meet the October 1, 1998, implementation date, went forward without addressing this recommendation. SPAS program officials told us that significant work remains to link the two systems. An ongoing FAA work group studying ways to incorporate system safety into Flight Standards’ programs has been given responsibility for linking ATOS and SPAS. This work group has not yet established a timetable for how or when the link will be completed. Page 18 GAO/RCED-99-183 FAA’s New Aviation Inspection System B-281502 Availability of ATOS Data The usefulness of ATOS data for analysis and targeting has been limited by Limits Analysis the overall lack of information in the ATOS database as well as by initial problems with accessing what information is available. FAA devoted much of the early part of fiscal year 1999 to planning ATOS inspections and training inspectors in the new system. As a result, very few data on the performance of the airlines covered by ATOS have been available for analysis. By the end of the second quarter, fewer than 60 of approximately 5,400 planned inspections had been completed, entered into the ATOS database, and made available to principal inspectors. These completed inspections comprised fewer than 1,200 individual inspection activities, substantially fewer than those recorded in previous years.12 The flow of inspection activity results into the ATOS database has since accelerated, with an additional 267 inspections and 1,808 individual inspection activities recorded over the first 6 weeks of the third quarter. Nevertheless, only 326 inspections, consisting of 3,079 inspection activities, had been completed by May 11, 1999, and for one airline no inspections had been completed. In contrast, during the first two quarters of fiscal year 1998, over 37,000 inspection activities had been recorded in the Program Reporting and Tracking System for the 10 airlines now covered by ATOS.13 We also found that access to the information available in the ATOS database was limited until inspections were completed. FAA designed ATOS so that neither principal inspectors nor analysts could access data until inspectors completed all inspection activities and the team’s data evaluation program manager reviewed and approved the data. The lack of access to key safety data created problems for principal inspectors, who are responsible for overseeing operations, maintenance, and avionics inspections at each airline. They could not view inspection results until the data evaluation program manager reviewed the data for clarity and consistency. Several principal inspectors expressed concern about not having timely access to key safety data, which they use as an early warning of potential safety risks. In February 1999, we briefed FAA on the concerns expressed about delayed access to inspection results in the ATOS database. In March 1999, FAA directed its automation contractor to take the steps necessary to grant principal inspectors immediate read-only access to inspection findings. Until this access was granted, principal inspectors remained dependent on 12 Under ATOS, each inspection recorded in the database summarizes the results of multiple observations called inspection activities. 13 While the 3,079 ATOS inspection activities may not be directly comparable to the 37,000 inspections conducted under the previous inspection system, a substantial drop-off in the inspection information available to managers and analysts has clearly occurred over the first half of fiscal year 1999. Page 19 GAO/RCED-99-183 FAA’s New Aviation Inspection System B-281502 such informal means as e-mail and telephone calls to learn of any problems or violations found during inspections. Principal inspectors now have access to all final inspection activity reports related to their specialty for the airline that they oversee. This change made data on 2,724 ATOS inspection activities available to principal inspectors as of May 11, 1999. ATOS’ Targeting The previously mentioned inadequacies in the ATOS guidance and problems Capabilities Are Limited by with the usefulness of the resulting data for analysis limit FAA’s ability to Inadequate Inspection use ATOS for targeting inspection resources to the problems that pose the greatest safety risks. Prior to the implementation of ATOS, the primary Guidance and Problems purpose of inspections was to identify individual safety problems and With the Usefulness of ensure their correction. Under ATOS, the primary objective is to provide Data reliable data to enable FAA to identify the highest-priority safety concerns and target the agency’s resources to reduce these concerns or risks as well as to ensure that individual safety problems are corrected. As implemented, ATOS falls short of this goal. Significant revisions will be needed to the ATOS guidance and database before the potential of this new inspection system can be realized. FAA Did Not Take Although FAA has many ongoing initiatives with the aviation industry, Advantage of Industry coordination with industry was lacking in the design of ATOS. Although Expertise in Developing several aspects of ATOS involve areas in which industry safety experts have experience and similar goals, the design process did not include airline or ATOS industry representatives, who were briefed on ATOS after the concept was developed. The industry’s input is missing, for instance, in the risk weights ATOS uses in its planning guidance to help determine the number of inspections a team is to conduct, thus directing resources to areas that require additional oversight. For example, the weight ATOS gives to an airline’s screening, boarding, and briefing procedures for passengers is greater than that assigned to cockpit procedures. FAA neither validated these weights nor discussed them with airline safety officials. We briefed FAA in December 1998 and again in February 1999 on the FAA Has Initiated problems we had found with the implementation of ATOS. Because of the Improvements to magnitude and seriousness of the problems associated with the current ATOS ATOS guidance and database, we suggested that FAA not expand ATOS to additional airlines, repair stations, or other aviation operations until these problems are resolved. Acknowledging that there were significant Page 20 GAO/RCED-99-183 FAA’s New Aviation Inspection System B-281502 challenges to further implementing the program, FAA agreed that expansion of the program needs to be delayed.14 In March 1999, the Director of Flight Standards and key ATOS program officials met with the principal inspectors who lead the 10 ATOS teams to brief them on our findings and to obtain their views on the system’s implementation. This meeting resulted in a list of immediate and future actions to be undertaken to address the concerns raised. First, FAA has taken steps to provide principal inspectors with immediate access to completed inspection reports. FAA also provided partial funding for planned overseas inspections, which will allow inspectors to complete some of the more critical overseas inspections. Finally, FAA will incorporate additional information on the inspection guidance task lists and database tracking system into ATOS training beginning in July. These initiatives will help alleviate several of the more immediate problems with ATOS by (1) making key safety information available to principal inspectors as soon as inspections are completed, (2) allowing at least some of the planned overseas inspections to take place, and (3) providing better training for inspectors on how to conduct inspections and record their results. In addition, both the ATOS Program Office and the ATOS internal audit group will continue to monitor the implementation closely to ensure that these problems and others that may arise are addressed. Funding has not yet been allocated to support the needed improvements to ATOS or to link ATOS with FAA’s existing data targeting and analysis system, SPAS. The ATOS concept offers significant promise for helping FAA overcome Conclusions deficiencies in its past approach to aviation safety inspections. However, its potential will not be fully realized until FAA resolves the problems resulting from the ambitious schedule it followed in implementing ATOS. These problems limit both FAA’s ability to conduct more systematic, structured inspections and analyze the resulting data to identify safety trends and its ability to target its resources to the greatest risks. The ATOS guidance is not clear and detailed enough to ensure more systematic, structured inspections that will result in more usable data. In addition, FAA has not adequately analyzed the data needs of ATOS users to ensure that the system collects the information that will enable the agency to perform critical trend and safety analyses. Such analyses are also limited because ATOS does not link to FAA’s other major database for safety analyses (SPAS). FAA has recognized the need for significant improvements before ATOS will 14 FAA already had plans to bring two additional airlines into ATOS in the near future. These airlines have recently completed certification activities and will begin ATOS inspections upon completion of a transition phase. Page 21 GAO/RCED-99-183 FAA’s New Aviation Inspection System B-281502 achieve its full potential as a system for overseeing commercial airline safety, and the agency has taken actions to correct some of the problems we found. Because of the challenges involved in making the needed improvements to ATOS, FAA officials have postponed plans to expand ATOS to other airlines until these problems are corrected. In addition, FAA is aware that the resolution of some problems with the implementation of ATOS involve broader issues that concern staffing decisions and workload issues that can affect all of Flight Standards’ inspectors. To strengthen the efforts to improve FAA’s aviation safety inspections and Recommendations the usefulness of the data that result from these inspections for analysis and for targeting the agency’s resources to the greatest potential safety threats, we recommend that the Secretary of Transportation direct the FAA Administrator to take the following actions: • Develop a structured process and timeline for working with inspectors to revise the Air Transportation Oversight System’s planning and inspection guidance. The process should involve the inspectors now using this guidance to (1) identify problems with the clarity of the guidance, (2) revise the inspection guidance to include tasks related to all applicable Federal Aviation Regulations, and (3) define the tasks to be completed during inspections. • Revise the inspection guidance to (1) include guidelines on the minimum number of times to perform various inspection tasks and (2) distinguish between tasks based on regulatory requirements and those based on handbook or other guidance. • Develop a plan that involves both inspectors and experts in risk assessment and database development in revising and refining the analysis of the data needs of users of the new inspection program. The requirements analysis should describe in detail the questions that need to be asked to improve safety, determine precisely what data are needed to answer those questions, and plan the appropriate analyses to be conducted on those data to answer the questions. • Restructure the inspection database to (1) require that essential data fields be completed before inspection reports can be closed out and (2) clearly indicate the proportion of inspection observations in which an airline complies with regulations. Page 22 GAO/RCED-99-183 FAA’s New Aviation Inspection System B-281502 • Determine what revisions will be needed to the Air Transportation Oversight System database and the agency’s existing Safety Performance Analysis System database to maximize the potential of these two systems by coordinating their trend analyses to identify potential safety risks. • Test and validate the revised guidance and database for the new inspection program. We provided FAA with a draft of this report for review and comment. We Agency Comments met with the Deputy Associate Administrator for Regulation and and Our Evaluation Certification, the Director of the Office of Flight Standards Service, the Acting Manager of the Air Transportation Oversight System Program Office, the Manager of the System Process Audit Group, and other FAA officials. The agency agreed with the substance of the report but commented that the tone was unnecessarily negative and could leave the impression that we believe that the program should be abandoned. Agency officials also said that they made a conscious decision to implement the new system aggressively, rather than in stages, and recognized that this approach would result in some implementation problems. However, they believe that ultimately the new system will be fully operational sooner than if they followed a more conservative implementation approach. The agency also commented that our review of the program was premature and suggested that an evaluation of the program in another year would find that most of the problems we reported had been resolved. We do not believe the program should be abandoned. We believe that our report clearly supports the Air Transportation Oversight System and acknowledges its potential for significantly strengthening FAA’s inspection process. However, we continue to believe that serious challenges need to be overcome before this program can achieve its potential. Because our review coincided with the program’s implementation, we were able to identify serious problems early and to promote constructive action by FAA to begin resolving them. As we reported, FAA has begun to address some of these problems. In commenting on this report, FAA also provided some updated information on its inspection activities and suggested wording revisions that we incorporated as appropriate. We conducted our work from September 1998 through June 1999 in accordance with generally accepted government auditing standards. Appendix II contains details of the scope and methodology of our review. Page 23 GAO/RCED-99-183 FAA’s New Aviation Inspection System B-281502 As you requested, unless you publicly announce its contents earlier, we plan no further distribution of this report until 10 days from the date of this letter. We will then send copies to the appropriate congressional committees; Rodney E. Slater, the Secretary of Transportation; Jane F. Garvey, the Administrator, FAA; Jacob J. Lew, the Director, Office of Management and Budget; and other interested parties. We will also make copies available to others upon request. If you have any questions about this report or need additional information, please call me at (202) 512-2834. Major contributors to this report are listed in appendix III. John H. Anderson, Jr. Director, Transportation Issues Page 24 GAO/RCED-99-183 FAA’s New Aviation Inspection System Page 25 GAO/RCED-99-183 FAA’s New Aviation Inspection System Contents Letter 1 Appendix I 28 The System Safety Assessment Tool 30 ATOS Guidance and The Air Carrier Assessment Tool 32 the Comprehensive The Comprehensive Surveillance Plan 33 Safety Attribute Inspections 33 Surveillance Plan Element Performance Inspections 34 Appendix II 35 Objectives, Scope, and Methodology Appendix III 37 GAO Contacts and Staff Acknowledgments Bibliography 38 Related GAO Products 39 Tables Table I.1: The Seven Airline Systems Defined in ATOS 30 Table I.2: Categories of System Safety Used With SSAT 31 Table I.3: ACAT Risk Indicators 32 Figure Figure I.1: How FAA Designs Comprehensive Surveillance Plans 29 Through ATOS Abbreviations ACAT air carrier assessment tool ATOS Air Transportation Oversight System EPI element performance inspection FAA Federal Aviation Administration GAO General Accounting Office SAI safety attribute inspection SPAS Safety Performance Analysis System SSAT system safety analysis tool Page 26 GAO/RCED-99-183 FAA’s New Aviation Inspection System Page 27 GAO/RCED-99-183 FAA’s New Aviation Inspection System Appendix I ATOS Guidance and the Comprehensive Surveillance Plan The Air Transportation Oversight System (ATOS) is a process designed to improve the Federal Aviation Administration’s (FAA) oversight of airlines. FAA’s Office of Flight Standards Service developed ATOS with the support of Sandia National Laboratories. ATOS uses system safety principles created for the nuclear industry and risk management to ensure that airlines have safety built into their operating systems. A systems safety approach means that FAA’s inspection efforts will cover all aspects of an airline’s performance that can affect safety and will focus on preventing accidents. A certificate management team oversees each of the 10 airlines under ATOS. Each team is led by three principal inspectors, one for each major area of inspections (operations, maintenance, and avionics). Additional team members include those based at the FAA office that holds the airline’s operating certificate and field inspectors in other FAA offices at locations to which the airline flies. The team uses automated planning guidance to develop a comprehensive surveillance plan for the airline. The planning guidance consists of two automated tools—the system safety analysis tool (SSAT) and the air carrier assessment tool (ACAT). The principal inspectors complete the SSAT and ACAT prior to an annual planning meeting. During the meeting, team members discuss the SSAT and ACAT, and their feedback is included in the final version. The results of the ACAT help define inspection activities that the team will include in the airline’s comprehensive surveillance plan. The SSAT, ACAT, and comprehensive surveillance plan are described in more detail below. Figure I.1 shows how these ATOS components relate to one another. Page 28 GAO/RCED-99-183 FAA’s New Aviation Inspection System Appendix I ATOS Guidance and the Comprehensive Surveillance Plan Figure I.1: How FAA Designs Comprehensive Surveillance Plans Through ATOS Source: Federal Aviation Administration. Page 29 GAO/RCED-99-183 FAA’s New Aviation Inspection System Appendix I ATOS Guidance and the Comprehensive Surveillance Plan Using the automated planning guidance, the team analyzes the airline’s operations, which are divided into seven systems (see table I.1). These systems are made up of 14 subsystems and of 79 elements. For example, the flight operations system is made up of two subsystems, which encompass elements such as aircraft dispatch and flight deck procedures. FAA designed the SSAT to help the team assess the systems in place at an airline to ensure safe operations. The ACAT applies a set of risk indicators to the airline’s subsystems and elements to generate a comprehensive surveillance plan. These risk indicators for the ACAT are based on safety and performance information that reflects areas of potential risk for an airline’s operations. Hence, the comprehensive surveillance plan will target those areas most likely to have safety problems. Table I.1: The Seven Airline Systems Defined in ATOS System System’s purpose Aircraft configuration and control Maintains the physical condition of the aircraft and associated components. Manuals Controls the information and instructions that define and govern an airline’s activities. Flight operations Governs aircraft movement. Personnel training and qualifications Ensures that an airline’s personnel are trained and qualified. Route structures Maintains an airline’s facilities on approved routes. Airman/crew member flight, rest, and duty Prescribes time limitations for airline time employees. Technical administration Addresses all other aspects of an airline’s certification and operations. Source: FAA Order 8400.10, Air Transportation Operations Inspector’s Handbook, appendix 6. The SSAT is a computerized tool designed to focus the inspection team’s The System Safety attention on the systems that an airline has in place. It poses questions to Assessment Tool the team covering six categories: safety attributes, safety culture, communications, accountability, training programs, and potential problem areas. (See table I.2.) The principal inspectors complete the SSAT prior to a yearly meeting to plan inspections to oversee the airline’s operations. To complete the SSAT, the principal inspectors rely on their knowledge of the airline and on the data available through FAA’s Safety Performance Analysis System (SPAS), the Flight Standards Automated System, or other sources. Before the annual meeting, the team members review the SSAT Page 30 GAO/RCED-99-183 FAA’s New Aviation Inspection System Appendix I ATOS Guidance and the Comprehensive Surveillance Plan completed by the principal inspectors and provide feedback. The SSAT is finalized at the annual meeting but may be revised during the year to retarget inspection resources. Table I.2: Categories of System Safety Used With SSAT Category Definition Safety attributes ATOS identifies six safety attributes: Responsibility: The unit or person in the airline that determines the course of action for a process. Authority: The unit or person in the airline that has the authority to establish or modify a process. Procedures: A documented method of accomplishing a process. Controls: A check or restraint that is designed into a process to ensure a desired result. Process measurements: The unit or person in the airline that measures and assesses information to identify, detect, analyze, and document problems or potential problems. Interfaces: Points at which independent processes interact. Safety culture The priority given to safety by the airline’s systems, including the airline’s identification and response to safety risks, and the effectiveness of internal evaluation systems. Communications The communication and feedback channels within the airline to report and respond to safety risks as well as open and timely communication with FAA and equipment manufacturers. Accountability The extent to which the airline holds its management and employees accountable for their assigned responsibility and authority. Training programs The priority an airline places on training as well as the effectiveness of initial and recurrent training programs. Potential problem areas The existence of concerns based on previous accidents or incidents, hotline complaints, or trends revealed in safety data. Source: FAA Order 8400.10, Air Transportation Operations Inspector’s Handbook, appendix 6. Page 31 GAO/RCED-99-183 FAA’s New Aviation Inspection System Appendix I ATOS Guidance and the Comprehensive Surveillance Plan By completing the SSAT, the ATOS inspection team assesses how well an airline addresses system safety issues. Using this information, the team determines whether to inspect any of the systems more or less frequently than suggested by the ATOS guidance and incorporates the inspections in the comprehensive surveillance plan for the airline. In completing the ACAT, principal inspectors use the results of the SSAT, The Air Carrier their knowledge of the airline their team oversees, and any other available Assessment Tool data to indicate concerns about any real or potential problem that could contribute to the failure of one of the airline’s elements, subsystems, or systems. The ACAT applies risk indicators to each of the airline’s systems. Table I.3 shows the types of risk indicators that are assessed when inspectors complete an ACAT. Table I.3: ACAT Risk Indicators Type of risk indicator Definition and examples Operational stability Those aspects of an airline’s organization and environment over which it has no direct control and that, when managed effectively, could enhance system safety and stability (e.g., turnover in personnel, or a merger or takeover). Airline dynamics Aspects of an airline’s environment that it directly controls and that could be used to enhance system safety and stability (e.g., an internal evaluation program, and risk management). Performance history The results of an airline’s operations over time (e.g., enforcement actions, self-disclosure reports to FAA). Environmental criticality Those aspects of an airline’s surroundings that could lead to or trigger a failure of one of its systems, subsystems, or elements and potentially create an unsafe condition (e.g., age of the fleet, outsourcing of maintenance). Source: FAA Order 8400.10, Air Transportation Operations Inspector’s Handbook, appendix 6. The principal inspectors complete the ACAT prior to the annual planning meeting. During the annual planning meeting, the team members provide feedback on the ACAT that is included in the final version. Page 32 GAO/RCED-99-183 FAA’s New Aviation Inspection System Appendix I ATOS Guidance and the Comprehensive Surveillance Plan The comprehensive surveillance plan is automatically generated based on The Comprehensive the information the team enters into ACAT. This provides a baseline Surveillance Plan surveillance plan that is tailored to the airline, reflecting concerns indicated by the principal inspectors. Each comprehensive surveillance plan incorporates two types of inspections, safety attribute inspections (SAI) and element performance inspections (EPI). SAIs appraise the quality of an airline’s safety attributes (see table I.2) for each system, its subsystems, and its elements. A team of inspectors conducts these system inspections. EPIs determine whether an airline adheres to its written procedures and controls for each system element and whether the established performance measures for each element are met. Individual inspectors conduct these inspections. ATOS allows the principal inspector to increase or, in some cases, decrease the level of inspection generated by the surveillance plan. This allows principal inspectors to use their expertise and personal knowledge of the airline to target resources toward the greatest safety risks. Although the comprehensive surveillance plan is automatically generated based on the results of the SSAT and ACAT, the plan is not finalized until the annual inspection planning meeting, which is attended by all members of the team. This allows the principal inspectors to discuss the completed SSAT and ACAT and to make changes based on other inspectors’ feedback. In addition, work assignments are discussed and made for each of the SAIs and EPIs that are planned. Principal inspectors complete and approve the final plan. The ACAT provides information directly to the SAI planning system in ATOS, Safety Attribute which indicates an inspection priority for each of the airline’s subsystems. Inspections Considering the SAI priority, a principal inspector enters the number of SAIs to be completed for each of the airline’s elements during the year. Automation of the SAI also allows a principal inspector to assign teams for each of the SAI activities. The principal inspector can also provide specific instructions to the team regarding the inspections. An SAI is an in-depth look at an airline’s policies and procedures for a system element. This inspection is structured to look at the safety attributes shown in table I.2. An SAI is completed by a team of inspectors, led by a team coordinator. This team assesses the accuracy and completeness of written policies and procedures governing each safety attribute associated with one of the airline system elements. For example, Page 33 GAO/RCED-99-183 FAA’s New Aviation Inspection System Appendix I ATOS Guidance and the Comprehensive Surveillance Plan one element of the route structure system is the line stations for servicing and maintaining the airline’s aircraft at each city it serves. Teams performing line station SAIs determine if there are qualified persons accountable for the line stations and if those persons have the authority to change the processes governing those facilities. The teams also review the procedures governing line stations to determine if proper controls are in place—such as standards for the maintenance conducted at the line station or for the training of line station employees. Finally, the teams determine if processes are in place to identify and correct problems as well as to ensure that other processes, such as de-icing and refueling, are coordinated. FAA established a baseline for how frequently each of the 79 ATOS elements Element Performance should be inspected. This frequency baseline identifies whether a system Inspections element should be inspected on an annual, semiannual, or quarterly basis within the planning cycle. The ACAT calculates an assessment value that, when applied to the frequency baseline, increases or decreases the number of inspections based on the concerns the principal inspectors have identified. Elements must be inspected at least once a year. Once the principal inspectors have determined the number of EPIs that will be conducted, the work is assigned to other inspectors on the team using the automated system. An EPI shows whether the airline follows the airline’s procedures and controls. Individual inspectors conduct EPIs, which most resemble the routine inspections FAA conducted in the past. For example, the line station EPI requires an inspector to visit a line station to determine if procedures and controls in place are being followed at that location. A line station inspection under ATOS may include multiple visits to one location or to a variety of other locations as well. All of these visits may be included in a single EPI report. Page 34 GAO/RCED-99-183 FAA’s New Aviation Inspection System Appendix II Objectives, Scope, and Methodology In September 1998, the Chairman and Ranking Democratic Member of the Subcommittee on Aviation, House Committee on Transportation and Infrastructure, asked us to address questions related to FAA’s new Air Transportation Oversight System: • To what extent does ATOS address past concerns about FAA’s aviation safety inspections? • What factors, if any, surfaced during the implementation of ATOS that could impede its success? • What is FAA doing to address any factors that could impede the success of ATOS? To determine to what extent ATOS addresses problems identified in the past with FAA’s inspection program, we reviewed previous reports by GAO, the Department of Transportation’s Inspector General, and internal FAA reports, such as FAA 90 Day Safety Review. In addition, we attended ATOS training provided to FAA inspectors and the annual inspection planning meeting held by 1 of FAA’s 10 certificate management teams. We interviewed members of the ATOS work group and program office to discuss how the new program was developed and the agency’s intentions for its implementation. In total, we interviewed 68 FAA employees assigned to the airlines ATOS covers, including 64 of 540 ATOS inspectors and 4 of the 10 unit supervisors. The 64 inspectors included 28 of the 30 principal inspectors who oversee ATOS airlines. In the two cases, we interviewed the assistant principal inspectors because the principal inspectors were not available. We discussed the ATOS concept, training, and implementation with each inspector. In addition to the 68 FAA employees assigned to oversee airlines under ATOS, we interviewed five Flight Standards district office managers and supervisors who oversee ATOS field inspectors to gain a broader perspective on inspector workload issues beyond those involving ATOS inspections. We also reviewed all 10 comprehensive surveillance plans developed for the 10 airlines covered by ATOS, as well as inspection findings reported through May 11, 1999. To determine what factors, if any, surfaced during the implementation of ATOS that could impede its success, we interviewed FAA’s principal inspectors for all 10 airlines covered by ATOS and also inspectors (including field inspectors) assigned to the certificate management teams. We interviewed staff from Sandia National Laboratories who served as consultants on the ATOS project and reviewed Sandia’s reports on the Page 35 GAO/RCED-99-183 FAA’s New Aviation Inspection System Appendix II Objectives, Scope, and Methodology system’s implementation. At FAA headquarters, we interviewed data management and system officials as well as staff assigned to the System Process Audit Group. We also reviewed reports prepared by these FAA units on the implementation of ATOS. In addition, we discussed ATOS specifics with key safety officials at the 10 airlines covered by the system. To determine what FAA is doing to address any factors that could impede the success of ATOS, we reviewed internal reports on ATOS and associated recommendations. We discussed these recommendations and what impact FAA’s budget shortfall will have on ATOS inspection efforts with principal inspectors on the ATOS teams and with Flight Standards headquarters officials. We conducted our work from September 1998 through June 1999 in accordance with generally accepted government auditing standards. Page 36 GAO/RCED-99-183 FAA’s New Aviation Inspection System Appendix III GAO Contacts and Staff Acknowledgments John H. Anderson, Jr., (202) 512-2834 GAO Contacts Robert White, (202) 512-5463 In addition to those named above, Leslie Albin, Bonnie A. Beckett, David Acknowledgments K. Hooper, Christopher M. Jones, Fran Featherston, Debra Prescott, and Stan Stenerson made key contributions to this report. Page 37 GAO/RCED-99-183 FAA’s New Aviation Inspection System Bibliography Booz-Allen & Hamilton Inc. Challenge 2000: Recommendations for Future Aviation Safety Regulation. Prepared for the Federal Aviation Administration. Apr. 1996. Federal Aviation Administration. FAA 90 Day Safety Review. Sept. 16, 1996. Sandia National Laboratories. “Analysis and Data Issues for the Implementation of a Systems Safety Focus in Air Transportation Oversight.” Oct. 6, 1998. U.S. Department of Transportation, Office of Inspector General. Audit of Aviation Inspection Program: Federal Aviation Administration (R6-FA-2-084). Washington, D.C.: May 29, 1992. U.S. Department of Transportation, Office of Inspector General. Final Report on Audit of Flight Standards Inspection Program in Southwest Region (R6-FA-9-078). Washington, D.C.: Mar. 17, 1989. U.S. Department of Transportation, Office of Inspector General. Pilot Examiner Program: Federal Aviation Administration (R2-FA-7-001). Washington, D.C.: Oct. 22, 1996. U.S. Department of Transportation, Office of Inspector General. Report on Audit of the Certification and Surveillance of Domestic and Foreign Repair Stations: Federal Aviation Administration (R4-FA-4-009). Washington, D.C.: Mar. 7, 1994. U.S. Department of Transportation, Office of Inspector General. Statement of Raymond J. DeCarli, Associate Deputy Inspector General, U.S. Department of Transportation. Washington, D.C.: May 21, 1997. White House Commission on Aviation Safety and Security. Final Report of the White House Commission on Aviation Safety and Security. Feb. 12, 1997. Page 38 GAO/RCED-99-183 FAA’s New Aviation Inspection System Related GAO Products Aviation Safety: Weaknesses in Inspection and Enforcement Limit FAA in Identifying and Responding to Risks (GAO/RCED-98-6, Feb. 27, 1998). Aviation Safety: FAA Oversight of Repair Stations Needs Improvement (GAO/RCED-98-21, Oct. 24, 1997). Aviation Safety and Security: Challenges to Implementing the Recommendations of the White House Commission on Aviation Safety and Security (GAO/T-RCED-97-90, Mar. 5, 1997). Aviation Safety: New Airlines Illustrate Long-Standing Problems in FAA’s Inspection Program (GAO/RCED-97-2, Oct. 17, 1996). Aviation Safety: Targeting and Training of FAA’s Safety Inspector Workforce (GAO/T-RCED-96-26, Apr. 30, 1996). Aviation Safety: Data Problems Threaten FAA Strides on Safety Analysis System (GAO/AIMD-95-27, Feb. 8, 1995). FAA Technical Training (GAO/RCED-94-296R, Sept. 26, 1994). Aviation Safety: FAA and the State Department Can Better Manage Foreign Enforcement Cases (GAO/RCED-94-87, Mar. 17, 1994). Aviation Safety: Progress on FAA Safety Indicators Program Slow and Challenges Remain (GAO/IMTEC-92-57, Aug. 31, 1992). Aviation Safety: FAA Needs to More Aggressively Manage Its Inspection Program (GAO/T-RCED-92-25, Feb. 6, 1992). Aviation Safety: Problems Persist in FAA’s Inspection Program (GAO/RCED-92-14, Nov. 20, 1991). Aviation Safety: Emergency Revocation Orders of Air Carrier Certificates (GAO/RCED-92-10, Oct. 17, 1991). Aviation Safety: FAA’s Safety Inspection Management System Lacks Adequate Oversight (GAO/RCED-90-36, Nov. 13, 1989). Aviation Training: FAA Aviation Safety Inspectors Are Not Receiving Needed Training (GAO/RCED-89-168, Sept. 14, 1989). Page 39 GAO/RCED-99-183 FAA’s New Aviation Inspection System Related GAO Products Aviation Safety: Needed Improvements in FAA’s Airline Inspection Program Are Underway (GAO/RCED-87-62, May 19, 1987). (348129) Page 40 GAO/RCED-99-183 FAA’s New Aviation Inspection System Ordering Information The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. VISA and MasterCard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent. Orders by mail: U.S. General Accounting Office P.O. Box 37050 Washington, DC 20013 or visit: Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000 or by using fax number (202) 512-6061, or TDD (202) 512-2537. Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone. A recorded menu will provide information on how to obtain these lists. For information on how to access GAO reports on the INTERNET, send an e-mail message with "info" in the body to: firstname.lastname@example.org or visit GAO’s World Wide Web Home Page at: http://www.gao.gov PRINTED ON RECYCLED PAPER United States Bulk Rate General Accounting Office Postage & Fees Paid Washington, D.C. 20548-0001 GAO Permit No. G100 Official Business Penalty for Private Use $300 Address Correction Requested
Aviation Safety: FAA's New Inspection System Offers Promise, but Problems Need to Be Addressed
Published by the Government Accountability Office on 1999-06-28.
Below is a raw (and likely hideous) rendition of the original report. (PDF)