United States General Accounting Office GAO Report to Congressional Requesters October 1997 AVIATION SAFETY FAA Oversight of Repair Stations Needs Improvement GAO/RCED-98-21 United States GAO General Accounting Office Washington, D.C. 20548 Resources, Community, and Economic Development Division B-274165 October 24, 1997 The Honorable Wendell H. Ford Ranking Minority Member Subcommittee on Aviation Committee on Commerce, Science, and Transportation United States Senate The Honorable Ron Wyden United States Senate As you requested, this report examines the Federal Aviation Administration’s (FAA) oversight of the aviation repair station industry. Specifically, this report addresses the following questions: (1) What is the nature and scope of the oversight of repair stations conducted by FAA personnel? (2) How well does FAA follow up on inspections to ensure that the deficiencies in repair station operations are corrected once they have been identified? (3) What steps has FAA taken to improve the oversight of repair stations? This report contains recommendations to the Secretary of Transportation for improving FAA’s oversight of repair stations. As you requested, unless you publicly announce its contents earlier, we plan no further distribution of this report until 30 days after the date of this letter. We will then send copies to the Secretary of Transportation; the Director, Office of Management and Budget; and other interested parties. We will make copies available to others upon request. If you or your staff have any questions, please call me at (202) 512-3650. Major contributors to this report are listed in appendix IV. Gerald L. Dillingham Associate Director, Transportation Issues Executive Summary U.S. airlines and air cargo companies now operate more than 6,700 Purpose aircraft, nearly 1,000 more than in 1990. Maintaining, repairing, and renovating this fleet costs about $6.5 billion a year. Nearly half of this work is now done by about 2,800 independent repair stations rather than by the air carriers themselves. Located worldwide, these repair stations vary greatly in size and scope. Some employ only a few people and fix a limited range of components, such as radios or instruments. Others have thousands of workers doing everything from conducting routine engine maintenance to rebuilding entire airframes. Although repair stations have been part of the aviation industry for decades, their use has grown substantially in recent years, particularly by airlines and cargo companies just entering the market. These new carriers have found it more economical to contract out much of their maintenance work rather than hiring their own staffs and building extensive facilities. Because repair stations deal with virtually all aircraft components, ensuring that their work is competently done is an important part of enhancing aviation safety. The Federal Aviation Administration (FAA) is the federal agency responsible for doing so. FAA conducts this oversight in two main ways: by sending its own inspectors to review repair station operations and by making airlines and air cargo companies responsible for ensuring that repair stations are following proper procedures. In recent years, FAA’s oversight of repair stations has become a matter of concern, in part because work performed by repair stations has been identified as a factor in several aircraft accidents. For example, the National Transportation Safety Board determined that the probable cause of an engine fire that destroyed a ValuJet DC-9 on an Atlanta runway in June 1995 was the inadequate procedures used by repair station personnel. The Ranking Minority Member of the Aviation Subcommittee of the Senate Committee on Commerce, Science, and Transportation, and Senator Ron Wyden asked GAO to examine FAA’s oversight of repair stations. GAO’s review focused on FAA’s own inspection activities at repair stations. At a later date, GAO plans to conduct a more detailed examination of FAA’s role in requiring airlines and air cargo companies to ensure that repair stations are following proper procedures. This current review examined the following questions: • What is the nature and scope of the oversight of repair stations conducted by FAA personnel? • How well does FAA follow up on inspections to ensure that the deficiencies in repair stations’ operations are corrected once they have been identified? Page 2 GAO/RCED-98-21 Aviation Safety Executive Summary • What steps has FAA taken to improve the oversight of repair stations? As the size of the airline and air cargo industries has grown, so has the Background reliance on repair stations. In 1990, repair stations performed 37 percent of air carriers’ maintenance; by 1996, the figure was 46 percent. More than 2,500 domestic and 270 foreign repair stations do work for air carriers. FAA’s certification process establishes what the repair stations are qualified to do. While many repair stations have fewer than 15 employees and a limited range of activities that FAA has certified, some employ thousands of workers who completely overhaul engines and renovate aging airframes for additional years of service. FAA’s inspection activities are aimed at ensuring that repair stations are still meeting the certification requirements. FAA had about 3,000 inspectors in fiscal year 1997. About 600 of them were involved in repair station inspections. FAA’s guidelines require that each repair station be inspected at least once a year. These inspections involve checking such matters as whether repair station staff have the appropriate qualifications to do the work and whether repair procedures meet FAA regulations. About 550 inspectors oversee repair stations located in the United States, usually through inspections conducted by individual inspectors. For larger facilities, these inspections may take place over several visits. Most of the inspectors are responsible for several repair stations as well as other types of operations, such as helicopter operators and training schools for pilots and mechanics. The remaining 50 inspectors inspect foreign repair stations that work on aircraft registered in the United States. These inspectors have fewer additional duties because FAA generally has no regulatory authority over foreign operations that do not directly affect aviation in the United States. Unlike their counterparts on the domestic side, these inspectors conduct most of their repair station inspections in teams. FAA’srecords indicate that the agency is meeting its goal of inspecting Results in Brief every repair station at least once a year. GAO examined FAA’s 1996 inspection records on about one-fourth of the 2,800 repair stations doing work for air carriers and confirmed that minimum inspection requirements had been met. In addition, 84 percent of the inspectors GAO surveyed stated that they believed the overall compliance of repair stations was good or excellent. However, more than half of the inspectors stated that there were areas of compliance that repair stations could improve. FAA Page 3 GAO/RCED-98-21 Aviation Safety Executive Summary relies primarily on reviews by individual inspectors of most domestic repair stations. In a few cases, FAA also uses teams to assess compliance at large, complex facilities. At such facilities, a team approach has been shown to be more effective at identifying problems than visits by individual inspectors, uncovering more systemic and long-standing deficiencies. A few of FAA’s offices have recognized that the traditional approach of relying on one inspector may be inadequate in such situations and have begun to use teams to inspect large repair stations. FAA officials acknowledge and support these initiatives. They said they believe these initiatives need to be evaluated and, if appropriate, used at other offices. GAO could not find sufficient documentation to determine how well FAA followed up to ensure that the deficiencies found during the inspections of repair stations were corrected. Thus, it was impossible to assess how completely or quickly repair stations were bringing themselves into compliance. FAA does not tell its inspectors what documentation to keep, and the resulting information gaps lessen the agency’s ability to determine how well its inspection activities are working or to identify and react to trends. These gaps in documentation are particularly important because FAA is spending more than $30 million to develop a reporting system that, among other things, is designed to use the documentation to make inspection decisions, such as where to apply the agency’s inspection resources to address those areas that pose the greatest risk to aviation safety. Following the May 1996 crash of a ValuJet DC-9 in the Florida Everglades, FAA announced new initiatives to upgrade the oversight of repair stations. These initiatives were directed at clarifying and augmenting air carriers’ oversight of repair stations, not at ways in which FAA’s own inspection resources could be better utilized. However, FAA does have three other efforts under way that would have a more direct bearing on its own inspection activities at repair stations. One effort would revise the regulations governing repair station operations, and another would revise the regulations governing the qualifications of repair station personnel. The revision of the repair station regulations began in 1989 and has been repeatedly delayed. The third effort is the addition of more FAA inspectors, which should mean that more resources can be devoted to inspecting repair stations. Page 4 GAO/RCED-98-21 Aviation Safety Executive Summary Principal Findings Current Inspection Most of FAA’s offices use the approach of assigning an individual inspector Approach Limits FAA’s to a repair station, even one that is large and complex, rather than Ability to Ensure assigning a team of inspectors. Although this one-inspector approach constitutes FAA’s primary frontline surveillance of repair stations, each Compliance at Large year regional and national decisions are made to use teams for more Repair Stations comprehensive reviews of a few repair stations. When direct comparisons could be made, teams were shown to be more effective than individual inspectors in identifying those areas in which repair stations were not in compliance with FAA’s rules and regulations, even if one inspector visited the facility several times and the team visited it just once. GAO reviewed 19 instances in which large repair stations inspected by one person had also been inspected by a special team during the same year. These special inspections are conducted at selected facilities that FAA regards as needing additional attention. The teams found a total of 347 deficiencies, only 15 of which had been identified in all of the visits made by individual inspectors in the year or more leading up to the special inspections. Deficiencies the teams identified included many that were systemic and apparently long-standing, such as inadequate training programs or poor manuals for quality control. Such deficiencies were likely to have been present when the repair stations were inspected earlier by individual inspectors. There are several reasons why team inspections identify a higher proportion of the deficiencies that may exist in the operation of large repair stations. Teams are better than individuals at ensuring that the inspection covers all areas of operations and that inspectors stay focused on the task at hand. Many FAA inspectors responsible for conducting inspections on their own said that because they have many competing demands on their time, their inspections of repair stations may not be as thorough as they would like. Another reason is that team inspections make greater use of checklists or other job aids for ensuring that all points are covered. FAA’s guidance requires inspectors to address all aspects of repair stations’ operations but does not prescribe any checklist or other means for specifying the items to be covered. The lack of a standardized approach increases the possibility that items will not be covered. Finally, inspectors believe team inspections help ensure that their judgments are independent because most team members have no ongoing relationship with the repair station. By contrast, individual-inspector reviews are Page 5 GAO/RCED-98-21 Aviation Safety Executive Summary conducted by personnel who have continuing regulatory responsibility for the facilities. A few of FAA’s offices have recognized that the traditional approach of relying on one inspector may be inadequate for overseeing the operations of large repair stations and have reconfigured their inspection resources to do more team inspections without adversely affecting other duties. They have done so mainly by redirecting the time formerly spent on reviews by individual inspectors into more systematic inspections done by a team of local, in-house staff. GAO identified FAA offices in Scottsdale, Arizona; Miami, Florida; and Seattle, Washington, as having initiated such changes on their own. FAA headquarters officials acknowledge and support these offices’ initiatives. They said they believe these initiatives need to be evaluated and, if appropriate, used at other offices. Follow-Up and FAA’s guidance is limited in specifying for inspectors what documents Documentation Need pertaining to inspections and follow-up need to be maintained in repair Attention station files. The closest thing to a requirement is a statement in the Airworthiness Inspector’s Handbook that the deficiency letter FAA sends to the repair station describing all deficiencies should be included in the repair station case file. GAO examined records of 172 instances in which FAA sent deficiency letters to domestic repair stations. The responses from the repair stations were not on file in about one-fourth of these instances, and FAA’s assessments of the adequacy of the corrective actions taken by the repair stations were not on file in about three-fourths of the instances. GAO also examined computer-based reports summarizing inspection information for FAA managers and found these reports were even less complete. Without complete documentation, it was impossible to assess how completely or quickly repair stations were bringing themselves into compliance. Better documentation is needed not only to allow FAA to demonstrate how quickly and thoroughly repair stations are complying with regulations, but also because it can affect FAA’s ability to identify performance trends involving the inspection of repair stations and to make informed decisions about them. FAA is spending more than $30 million to develop a reporting system that, among other things, is designed to use this documentation to make decisions on applying inspection resources to those areas posing the greatest risk to aviation safety. Such a system will be of limited use if the documentation on which it is based is inaccurate, incomplete, or outdated. FAA must have data to show where safety problems and deficiencies exist Page 6 GAO/RCED-98-21 Aviation Safety Executive Summary and, thus, where to better target its limited inspection resources. In 1995, as part of a prior study examining FAA’s information management systems, GAO recommended that FAA develop a comprehensive strategy for making data-related improvements. FAA agreed, but it fell behind in its schedule for making improvements. Continued monitoring will be needed to ensure that the actions taken are sufficient to resolve the problems by December 1999, when the new reporting system is scheduled for completion. Documentation of inspections and follow-up was better in FAA’s files for foreign repair stations, perhaps in part because under FAA regulations, foreign repair stations must renew their certification every 2 years. By comparison, domestic repair stations retain their certification indefinitely unless they surrender it or FAA suspends or revokes it. Foreign repair stations appear to be correcting their deficiencies quickly so that they qualify for certificate renewal. The 34 FAA inspectors GAO interviewed who had conducted inspections of both foreign and domestic repair stations were unanimous in concluding that compliance occurred more quickly at foreign facilities. They attributed the quicker compliance to the renewal requirement and said that it allowed them to spend less time on follow-up, freeing them for other surveillance work. However, because of the poor documentation in domestic repair station files, GAO was unable to confirm whether foreign repair stations achieve compliance more quickly than domestic repair stations do. Actions Under Way The six repair station initiatives announced in June 1996 by the previous Directed Primarily at Air FAA Administrator following the ValuJet crash are directed at clarifying Carriers’ Oversight of and augmenting air carriers’ responsibilities for overseeing repair stations. For example, one initiative requires that before an air carrier can add a Repair Stations repair station to the list of repair stations doing substantial maintenance on its aircraft, the carrier must conduct an audit to verify that the repair station is capable of doing the work in accordance with the carrier’s approved programs. GAO did not directly assess the initiatives in this review because the initiatives are not focused on strengthening FAA’s own inspection and follow-up efforts. FAA inspectors assigned to oversee repair stations told GAO that the initiatives would have no effect on their direct inspections of repair stations. Several other efforts unrelated to the June 1996 initiatives may hold potential for improving FAA’s own inspections of repair stations. Two involve initiatives to change the regulations covering repair station Page 7 GAO/RCED-98-21 Aviation Safety Executive Summary operations and the certification requirements for mechanics and repairmen. FAA acknowledges that the existing regulations do not reflect many of the technological changes that have occurred in the aviation industry in recent years. The FAA inspectors surveyed by GAO strongly supported a comprehensive update of repair station regulations as a way to improve repair stations’ compliance. This update began in 1989, has been repeatedly delayed, and still remains in process. The most recent target—to have draft regulations for comment published in the Federal Register during summer 1997—was not met. Similarly, the update of the certification requirements for maintenance personnel has been suspended since 1994. Because of these long-standing delays, completion of both updates may require additional attention on management’s part to help keep both efforts on track. The third effort involves increasing FAA’s inspection resources: Since fiscal year 1995, FAA has been in the process of adding more than 700 inspectors to its workforce who will, in part, oversee repair stations. Survey responses from current inspectors indicated that the success of this effort will depend partly on the qualifications of the new inspectors and on the training available to all those in the inspector ranks. To improve FAA’s oversight of repair stations, GAO recommends that the Recommendations Secretary of Transportation instruct the Administrator of FAA to take the following actions: • Expand the use of locally based teams for repair station inspections, particularly for repair stations that are large, complex, have higher rates of noncompliance, or meet predetermined risk indicators; and develop and use checklists or job aids for inspectors as a way of bringing about more comprehensiveness and standardization. • Specify what documentation should be kept in repair station files to record complete inspection results and follow-up actions. • Monitor the implementation of the strategy for improving the quality of the data to be used in FAA’s new management information system. • Expedite the efforts to update regulations pertaining to the oversight of repair stations, and establish and meet schedules for completing the updates. GAO provided the Department of Transportation and FAA with a draft of this Agency Comments report for their review and comment. GAO met with FAA officials, including the Deputy Associate Administrator for Regulation and Certification Page 8 GAO/RCED-98-21 Aviation Safety Executive Summary (acting on behalf of the FAA Administrator) to obtain FAA’s comments. FAA agreed with the draft’s overall message and recommendations. FAA said it will build on its already successful repair station inspection program to enhance the oversight of this sector of the aviation industry. FAA cited several agency initiatives that it said are under way to do that. FAA’s 90-day safety review conducted last year recommended the creation of an analytic unit that could provide safety trend data to inspectors. FAA said an office within the Flight Standards Service was created on May 20, 1997, to provide data that will help focus inspection and other resources. The review also recommended that field and division managers be given the flexibility to determine the skills needed in a particular field office to ensure the appropriate mix of technical, paratechnical, support, and clerical expertise. FAA said that this flexibility will be supported through the establishment of new staffing standards—a long-term project that is already under way. Additionally, FAA provided some technical comments and corrections on the draft report. GAO revised the report as appropriate to reflect FAA’s technical comments and corrections. Page 9 GAO/RCED-98-21 Aviation Safety Contents Executive Summary 2 Chapter 1 14 Growing Air Carrier Industry Creates Additional Need for 14 Background Maintenance and Repair Services What Are Repair Stations? 15 How Does FAA Oversee Repair Stations? 19 Who Else Has Oversight Responsibility for Repair Stations? 21 Why Has FAA’s Oversight of Repair Stations Been a Matter of 22 Concern? Objectives, Scope, and Methodology 25 Chapter 2 27 Number of Inspections Conducted Meets the Required Minimum 27 Current Inspection Team Inspections Provide More Detailed Review Than Those 30 Approach Limits Conducted by Individual Inspectors Different Approaches to Repair Station Surveillance Show 39 FAA’s Ability to Promise Ensure Compliance Conclusions 47 Recommendations 47 Agency Comments 48 Chapter 3 49 Information on Inspection Results Is Important but Incomplete 49 Incomplete Compliance May Come More Quickly at Foreign Repair Stations 56 Documentation of Conclusions 58 Recommendations 59 Inspections Makes Agency Comments 59 Extent of Follow-Up Difficult to Determine Chapter 4 60 Existing Initiatives Are Targeted Primarily at Air Carriers 60 Actions Currently Planned Update of Repair Station Regulations Is Slow in Coming 65 Under Way to Adding to the Inspector Corps and Improving Training 68 Upgrading Certification and Training of Aviation Maintenance 70 Augment Oversight of Personnel Repair Stations Conclusions 71 Recommendation 71 Agency Comments 71 Page 10 GAO/RCED-98-21 Aviation Safety Contents Appendixes Appendix I: Objectives, Scope, and Methodology 72 Appendix II: Survey Methodology 77 Appendix III: Survey Questions and Responses 81 Appendix IV: Major Contributors to This Report 91 Tables Table 1.1: Repair Station Rating Categories 15 Table 1.2: Comparison of FAA’s Oversight of Domestic and 20 Foreign Repair Stations Table 1.3: Recent Accidents Involving Aircraft Maintained by 24 Contract Repair Stations Table 2.1: Characteristics of a Quality Inspection 32 Table I.1: Airlines Included in GAO’s Detailed Review 73 Table I.2: Repair Stations Included in Detailed Review 74 Table I.3: Distribution of Repair Stations Overseen by FAA 75 Inspectors, by Size Table II.1: Sampling Errors for Selected Percentages 78 Table II.2: Sampling Errors for Repair Stations by Number of 78 Employees Table II.3: Summary of Returns to Mail Surveys 80 Figures Figure 1.1: Types of Maintenance Performed by Repair Stations 16 Figure 1.2: Maintenance Being Done on a Boeing 727 at Tramco, 18 Inc., Everett, Washington Figure 1.3: Overview of the Oversight Given Repair Stations 22 Figure 2.1: Types of Deficiencies FAA Teams Identified During 31 Special, In-Depth Inspections Figure 2.2: FAA Inspectors’ Responses on Barriers to 34 Comprehensive Inspections Figure 2.3: FAA Inspectors’ Responses on Barriers to Focusing 35 During Inspections Figure 2.4: FAA Inspectors’ Views on the Current Overall 37 Compliance of Repair Stations Figure 2.5: FAA Inspectors’ Responses on Amount of 39 Improvement Needed for Repair Station Compliance Figure 2.6: FAA Inspectors’ Responses on Using Teams to 45 Improve Compliance Figure 3.1: Extent of Follow-Up Documentation for Inspections 52 of Selected Domestic Repair Stations, Fiscal Years 1993 Through 1996 Page 11 GAO/RCED-98-21 Aviation Safety Contents Figure 3.2: Extent of Follow-Up Documentation for Selected 53 Foreign Repair Station Inspections, in Fiscal Years 1993 Through 1996 Figure 4.1: June 1996 Initiatives Designed to Improve Air Carrier 62 Oversight of Repair Stations Figure 4.2: FAA Inspectors’ Responses to Ideas for Improving 67 Compliance Through Changes Related to Existing Regulations Figure 4.3: FAA Inspectors’ Responses to Ideas for Improving 69 Compliance Through Inspector Qualifications Abbreviations FAA Federal Aviation Administration JAA Joint Aviation Authorities NASIP National Aviation Safety Inspection Program PTRS Program Tracking and Reporting Subsystem RASIP Regional Aviation Inspection Program SPAS Safety Performance Analysis System Page 12 GAO/RCED-98-21 Aviation Safety Page 13 GAO/RCED-98-21 Aviation Safety Chapter 1 Background Passenger airlines, air freight companies, and other air carriers in the United States spend almost $6.5 billion every year maintaining and repairing their aircraft, according to industry estimates. While these carriers have traditionally performed much of this maintenance and repair work themselves, many are now contracting an increasing portion of the work to about 2,800 repair stations in the United States and other countries.1 As the agency responsible for overseeing the aviation industry, the Federal Aviation Administration (FAA) has the primary responsibility for ensuring that repair stations are operating in accordance with laws and regulations. Commercial air carriers certified in the United States now operate more Growing Air Carrier than 6,700 aircraft, nearly 1,000 more than in 1990. Operators include more Industry Creates than 150 airlines, freight carriers, charter firms, and other companies Additional Need for certified by FAA and operating under part 121 of the Federal Aviation Regulations.2 The aircraft they operate range from planes such as a Maintenance and Fairchild Metroliner III, which typically carries a maximum of 19 Repair Services passengers or about 5,000 pounds of cargo, to planes such as a Boeing 747-400, which is capable of carrying more than 400 passengers or 122 tons of cargo. Some of the largest companies, like United Airlines or American Airlines, may have 500 or more aircraft. With more aircraft flying, the need for maintenance and repair services has grown. Air carriers spent almost $6.5 billion for maintenance and repair on their aircraft in 1996, according to an industry estimate.3 This amount is an increase of $1.2 billion, or 23 percent, over the estimate of $5.3 billion in 1990. The term “maintenance and repair” encompasses a wide variety of activities. Some activities involve frequent servicing, such as overhauling tires, wheels, and brakes. Others include more extensive renovation, such as airframe maintenance, that must be done as aircraft get older. FAA classifies maintenance and repair activities into six rating categories (see 1 Although nearly 5,000 repair stations are certified by FAA, data provided in FAA’s Vital Information Subsystem specifically identified about 2,800 of those as performing work on aircraft with 10 or more seats. Of these facilities, more than 2,500 are in the United States, and 273 are in foreign countries. This report addresses the oversight of these 2,800 repair stations. 2 Air carriers now operating under part 121 use aircraft configured for 10 or more passengers. New rules adopted by the FAA in 1995 require certain commuter operators conducting scheduled operations under part 135 to conduct those operations under part 121 beginning in March 1997. Included were those air carriers conducting scheduled operations carrying passengers with aircraft configured for 10 to 30 seats. This report uses the term “air carriers” to refer to companies operating under part 121, including those that formerly operated under part 135. 3 This figure includes air carriers with revenues exceeding $100 million annually. Page 14 GAO/RCED-98-21 Aviation Safety Chapter 1 Background table 1.1), which it uses to designate the type of maintenance or activity it has certified a repair station to perform. Table 1.1: Repair Station Rating Categories Ratinga Examples of service performed Accessory Functional check and calibration of fuel control unit Airframe Inspection and repair for corrosion and fatigue damage Instrument Calibration of air speed indicator Power plant Borescope inspection of internal engine components Propeller Examinations and repair of cracks, nicks, and deformations Radio Measurement of frequency and power of transmitting unit a FAA also issues limited ratings for items such as nondestructive testing, maintenance on emergency equipment or landing gear, or other specialized services not included in the aircraft rating categories. Some major air carriers, such as American Airlines and United Airlines, have substantial maintenance facilities of their own. However, many air carriers, including smaller air carriers, have used third-party repair stations rather than invest in the additional staff and hardware needed to do the work in-house. Some new air carriers entering the passenger or air freight markets have chosen to rely heavily—and in some cases, almost exclusively—on repair stations. The term “repair station” spans a wide variety of operations. In 1996, there What Are Repair were almost 5,000 repair stations certified by FAA, about 2,800 of which Stations? performed maintenance work on aircraft used by air carriers.4 A repair station’s certificate specifies the types of maintenance it can perform. Some repair stations specialize in one particular maintenance and repair category, while others may conduct work in several categories. As figure 1.1 shows, the types of maintenance most often included in the certificates of these 2,800 repair stations were for accessories and airframes. 4 The other 2,200 certified repair stations worked on general aviation aircraft which are regulated under part 91 of the Federal Aviation Regulations. General aviation comprises all civil aircraft operations except those involving such commercial activities as the transport of revenue-paying passengers. Over 90 percent of the aircraft registered in the United States are general aviation aircraft. Page 15 GAO/RCED-98-21 Aviation Safety Chapter 1 Background Figure 1.1: Types of Maintenance Performed by Repair Stations FAA-certified repair stations performing work for air carriers: Type of maintenance 2,800 covered in certification Accessories Airframes 56% Power plants All other Instruments FAA-certified 44% repair stations: Radio 2,200 Propellers 0 200 400 600 800 1,000 1,200 Number of repair stations Note: Some repair stations are certified for more than one type of maintenance. Source: FAA data. In addition to specifying the types of maintenance a repair station can perform, FAA may limit the scope of a repair station’s activities. For example, whenever appropriate, FAA may issue a rating that limits a repair station’s work to maintaining or altering only certain types of airframes, power plants, propellers, radios, instruments, or accessories. Such a rating may be limited to a specific model of aircraft, engine, or constituent part or to any number of parts made by a particular manufacturer. FAA also issues limited ratings for specialized work, such as nondestructive testing, maintenance on landing gear or emergency equipment, or other specific areas not included in any of the six standard rating categories.5 5 As of Nov. 15, 1996, 251 repair stations performing work for air carriers held one or more limited ratings for items such as nondestructive testing, maintenance on emergency equipment or landing gear, or other specialized service or maintenance not included in the aircraft rating categories. Page 16 GAO/RCED-98-21 Aviation Safety Chapter 1 Background Repair stations vary considerably in size and scope of operations. For example, Tramco, Inc., located in Everett, Washington, is one of the largest repair stations in the United States, with hangar facilities of 450,200 square feet and a workforce of more than 2,000. At one time, this facility can accommodate five wide-body aircraft, such as Boeing 747s, and five narrow-body aircraft, such as Boeing 737s. The repair station primarily conducts regularly scheduled maintenance and modifications, and it also modifies new aircraft when specifications are changed after manufacturing is completed. Figure 1.2 shows maintenance being done on a Boeing 727 at this facility. By contrast, Precision Avionics & Instruments in Atlanta, Georgia, is a much smaller repair station. It employs 35 workers and has a facility of 24,000 square feet where it primarily services instruments, electrical and electronic components and accessories. Page 17 GAO/RCED-98-21 Aviation Safety Chapter 1 Background Figure 1.2: Maintenance Being Done on a Boeing 727 at Tramco, Inc., Everett, Washington While most domestic repair stations are operated independently of commercial airlines, a few are in-house maintenance operations that conduct work for other airlines on a contractual basis. For example, at its own maintenance facilities, Delta Airlines performs power plant maintenance for such carriers as American Airlines, Air Jamaica, Trade Winds, and Aeroflot Russian International Airlines. Repair stations that work on the aircraft of U.S. carriers are found throughout the rest of the world, though not in as great a number as repair Page 18 GAO/RCED-98-21 Aviation Safety Chapter 1 Background stations in the United States. In all, about 270 FAA-certified foreign repair stations perform repair work for U.S. air carriers. For example, Sabena Technic, the maintenance arm of Sabena Belgian World Airlines, does engine repair work for Federal Express and other carriers at its facility in Brussels. Sabena has FAA’s approval for work on airframes, power plants, radios, instruments, and accessories. FAA’s oversight of repair stations is divided into two phases—certification How Does FAA and surveillance. Certification initially involves a repair station’s applying Oversee Repair to FAA for authority to perform certain types of maintenance on certain Stations? types of aircraft. FAA inspects the repair station to ensure that the applicant’s proposed procedures are effective and that the equipment meets regulatory requirements. In addition, FAA also inspects facilities, personnel, and material as well as the repair station’s inspection system. If FAA finds these things to be in order, it issues a certificate with a set of “operation specifications” that cover what maintenance activities the repair station is authorized to perform. Certification is handled in one of two ways, depending on whether the repair station is in the United States or abroad. FAA requires foreign repair stations to renew their certification at least every 2 years, but for domestic repair stations, certification is permanent unless it is surrendered by the applicant or suspended or revoked by FAA.6 Surveillance, usually in the form of inspections, follows certification. FAA’s guidelines require its safety inspectors to perform a facility inspection of each domestic and foreign repair station at least once every year. For many of the larger domestic repair stations, this inspection is broken into multiple visits. For example, FAA inspectors visited Evergreen Air Center, one of the larger repair stations we reviewed in depth, more than 20 times during fiscal year 1996. Located in Marana, Arizona, Evergreen employs about 590 workers who conduct all types of maintenance on most types of large transport aircraft. FAA divides repair station inspections into two categories, avionics and maintenance. Avionics inspections focus on a repair station’s overall program for aircraft electronic components, including personnel training, policies, and procedures. Maintenance inspections cover a repair station’s overall maintenance program, including personnel training, policies, and procedures. 6 Repair stations are regulated under 14 C.F.R. part 145. Specifically, under FAA regulations, a foreign repair station’s certificate, or rating, expires 12 months after the date on which it was issued, unless it is surrendered, suspended, or revoked before that time. FAA can make subsequent renewals for periods of up to 24 months. Page 19 GAO/RCED-98-21 Aviation Safety Chapter 1 Background FAA’s certification and inspection activities are carried out by inspectors based in the United States and abroad (see table 1.2).7 On the domestic side, certification and inspection activities are carried out by more than 550 FAA inspectors, most of whom have many other responsibilities as well. Unless they are assigned to one of the largest operations, inspectors usually are responsible for more than one repair station. We examined the workloads of 98 inspectors at the FAA offices we visited and found that the number of repair stations these inspectors were responsible for ranged from 1 to 42, with a median workload of 12 repair stations. These repair stations varied in size and complexity. Most of the inspectors had many other surveillance responsibilities as well, such as overseeing training schools for pilots and mechanics, helicopter operators, agricultural operators, and air taxis. On the foreign side, about 50 FAA inspectors handle the oversight of repair stations, again with responsibility for multiple repair stations. Unlike their counterparts in the United States, however, inspectors in these offices generally have the oversight of repair stations as their primary responsibility. Table 1.2: Comparison of FAA’s Oversight of Domestic and Foreign Domestic Foreign Repair Stations Number of FAA principal 552 49 inspectors assigned to oversee repair stations as of Nov. 15, 1996 Location of FAA principal 86 FAA offices throughout 7 FAA offices—Brussels, inspectors assigned to the United States Dallas/Fort Worth, Frankfurt, oversee repair stations London, Miami, San Francisco, and Singapore Number of repair stations 2,504a 273a inspectors are responsible for Responsibility for oversight of Varies with other oversight Primary responsibility repair stations duties a These numbers only include those repair stations identified in FAA’s Vital Information Subsystem as performing work on aircraft with 10 or more seats. 7 In addition to inspectors with direct oversight, other FAA inspectors may also visit repair stations. An inspector responsible for an air carrier that contracts with a repair station may also review the repair station’s operations, but only insofar as they pertain to the work being done for the air carrier. If an inspector is not based near a repair station, he or she may request that another inspector—called a geographic inspector—close to the facility make the visit. Page 20 GAO/RCED-98-21 Aviation Safety Chapter 1 Background Under Federal Aviation Regulations, air carriers must ensure that repair Who Else Has stations are conducting work that conforms with the air carriers’ manuals Oversight and the applicable FAA regulations.8 As part of meeting this requirement, Responsibility for air carriers may use one or both of the following means: Repair Stations? • They may conduct their own audits of repair stations—generally every 2 years—to ensure that the facilities have the capability to perform the work in accordance with the air carriers’ maintenance policies, procedures, and requirements. • They may rely on audits conducted by the Coordinating Agency for Supplier Evaluation, an international industry organization of major airlines and aerospace and marine contractors. These audits are conducted—again, generally every 2 years—by staff from member airlines who use a standardized approach that includes Federal Aviation Regulation requirements. Because many airlines use the same repair stations, these audits eliminate the expense of redundant evaluations of repair stations. Repair stations, both foreign and domestic, are also potentially subject to review by the regulatory agencies of other countries. Many of the national aviation authorities in countries where repair stations are located have developed their own extensive inspection, surveillance, evaluation, and certification programs for repair facilities. Like FAA, many of these agencies review repair stations in other countries as well (including the United States). Twenty-seven European nations have banded together to coordinate their efforts through an organization called the Joint Aviation Authorities (JAA), but many nations such as China conduct reviews on their own.9 Like FAA, these other regulators have set up their programs to help ensure compliance with their own national standards. Figure 1.3 summarizes the relationship of the various parties involved in the oversight of repair stations. 8 See subpart L of part 121 and subpart J of part 135. 9 Under the provisions of the Bilateral Aviation Safety Agreement, FAA is negotiating maintenance implementation procedures with European countries that are members of JAA to provide reciprocal acceptance of surveillance information on FAA-certified repair stations in Europe, and JAA-approved maintenance organizations in the United States. We did not address these negotiations in our review because discussions are still largely in the formative stage and little or no near-term effect is anticipated. Page 21 GAO/RCED-98-21 Aviation Safety Chapter 1 Background Figure 1.3: Overview of the Oversight Given Repair Stations Federal Aviation Air Carriers Other Countries Administration (if applicable) Certification Surveillance Audits National requirements After receiving repair All repair stations are Air carriers must ensure U.S. repair stations may station's application, FAA inspected by FAA at least that repair stations can receive oversight from reviews equipment, annually conform with the air other countries if parts procedures, and personnel carriers' policies, or products they repair to ensure regulatory procedures, and are operated in that requirements are met FAA inspectors overseeing air carriers requirements country that contract with the FAA issues "operations repair station may also Audits can be conducted Foreign repair stations specification" listing tasks review repair station by individual air carriers or may be reviewed by and aircraft for which operations by an industry association their own country's repair station is approved regulatory agency When FAA inspectors Foreign repair stations find deficiencies, repair must be recertified at least stations are responsible every 2 years; domestic for bringing operations repair stations face no into compliance recertification requirement Concern has arisen about FAA’s oversight of repair stations for three Why Has FAA’s reasons: Air carriers are relying on repair stations much more than in the Oversight of Repair past. Several recent accidents have involved aircraft maintained by repair Stations Been a stations. And FAA’s oversight of repair stations is comparatively limited. Matter of Concern? Steady growth in air carriers’ use of repair stations is one development that has focused additional attention on how FAA is carrying out its responsibility to oversee repair stations. Reliance on repair stations among air carriers has grown from an estimated 37 percent of total maintenance in 1990 to an estimated 46 percent in 1996. Page 22 GAO/RCED-98-21 Aviation Safety Chapter 1 Background Reliance on repair stations has been particularly heavy among newer carriers such as ValuJet, Western Pacific, Reno Air, and Frontier Airlines. According to FAA officials with whom we spoke, newer carriers use repair stations extensively because they do not have enough repair work to make performing it themselves economical or because they want to ensure that they get an experienced cadre of mechanics with sound practices and procedures. For example, Reno Air uses AAR Oklahoma, Inc., to perform heavy airframe maintenance and major alterations of its MD-80s and MD-90s. Operating only 30 of these aircraft does not warrant Reno Air’s investing in the in-house repair capabilities for this type of maintenance. And even though established air carriers tend to use repair stations less extensively than smaller, newer air carriers, the amount of maintenance they conduct is so great that if only a small percentage of their maintenance is performed at repair stations, it still represents a substantial amount of work. For example, a United Airlines official estimated that while the company contracts out only about 7 percent of its maintenance budget to repair stations, this amounted to about $126 million worth of work in 1996. A second, and significant, reason for concern about FAA’s oversight of repair stations stemmed from domestic aviation accidents in 1995 and 1996. Table 1.3 describes four aviation accidents for which the National Transportation Safety Board found contributing factors that involved inadequate inspection or maintenance or improper handling of hazardous cargo by repair stations. Page 23 GAO/RCED-98-21 Aviation Safety Chapter 1 Background Table 1.3: Recent Accidents Involving Aircraft Maintained by Contract Repair Link to repair station Stations Airline, aircraft, and date Nature of accident activity ValuJet DC-9, June 8, 1995 During a takeoff at Atlanta, The National Transportation an uncontained engine Safety Board (the Board) failure caused a fire that determined that the destroyed the aircraft’s probable cause of the fire fuselage. Several people was the failure of repair were injured during the station personnel to evacuation. conduct a proper inspection of the engine assembly. Atlantic Southeast Airlines The aircraft lost a propeller The Board determined that Embraer-120, August 21, while climbing above the probable cause of the 1995 18,100 feet. It crashed loss of the propeller was a during an emergency fatigue fracture from landing, killing 8 and corrosion pits that were not injuring 21 others on board. discovered or properly repaired by the manufacturer’s repair station. ValuJet DC-9, May 11, 1996 A fire broke out in a cargo In an abstract of the final compartment of the aircraft, report, the Board said a which crashed in the Florida probable cause of the Everglades, killing all 110 accident was the failure of a on board. repair station to properly prepare, package, identify, and track unexpended oxygen generators, a hazardous material. Tower Air The aircraft sustained minor The Board found problems Boeing 747-136, June 17, damage when an engine associated with a repair 1996 accessory gearbox caught station’s overhaul and fire during the descent for assembly of a drive unit. landing. No one was injured. Source: National Transportation Safety Board. A third reason for concern is the relatively limited amount of oversight that FAA gives repair stations compared with the oversight it gives air carriers. FAA is responsible for ensuring that repair stations comply with regulations, and the agency’s annual guidance for surveillance sets forth minimum inspection requirements for all certificate holders. For fiscal year 1997, each repair station was to have a minimum of one facility inspection, while each air carrier was required to have many more inspections. An air carrier such as Alaska Airlines, for instance, had to have a minimum of 62 inspections. Page 24 GAO/RCED-98-21 Aviation Safety Chapter 1 Background The Ranking Minority Member of the Aviation Subcommittee of the Senate Objectives, Scope, Committee on Commerce, Science, and Transportation, and Senator Ron and Methodology Wyden asked us to examine FAA’s oversight of repair stations. Specifically, we were asked to address the following questions: • What is the nature and scope of the oversight of repair stations conducted by FAA personnel? • How well does FAA follow up on inspections to ensure that deficiencies in repair stations’ operations are corrected once they have been identified? • What steps has FAA taken to improve the oversight of repair stations? Our analysis was based in part on agencywide data FAA provided and in part on a detailed review of a cross-section of airlines, repair stations, FAA offices, and FAA inspectors. In general, we did the following: • We reviewed the use of repair stations by eight air carriers, chosen because, like the industry as a whole, they varied greatly in the extent to which they used repair stations.10 The number of aircraft operated by these carriers ranged from 3 to 659. • We reviewed operations at 10 repair stations, chosen because they represented a wide variety of locations (both domestic and foreign), types of repair activities, and size of operations.11 The repair stations ranged from a wheel and brake specialist with about 20 employees to facilities conducting many types of maintenance and employing more than 3,000 workers. • We examined oversight activities and discussed the oversight of repair stations at FAA headquarters, 4 of FAA’s 9 regional offices, 8 of FAA’s 86 Flight Standards district offices, and 6 of FAA’s 7 offices with international responsibilities.12 Our work at these offices included reviewing inspection files for nearly 500 repair stations. • We conducted a survey of 275 FAA principal inspectors on their views about ways to improve the oversight of repair stations. Our survey had a 10 The air carriers were Alaska, American, America West, Delta, Sierra Pacific, Simmons, Southwest, and United. 11 The repair stations were Advanced Material Technologies, Inc., Tempe, Arizona; AeroControls, Inc., Auburn, Washington; B.F. Goodrich Component Services Division, Tempe, Arizona; Chromalloy Los Angeles, Gardena, California; Evergreen Air Center, Marana, Arizona; Greenwich Air Services, Inc., Miami, Florida; Lufthansa Technik AG, Frankfurt, Germany; Precision Avionics and Instruments, Inc., Atlanta, Georgia; Sabena Technic, Brussels, Belgium; and Tramco, Inc., Everett, Washington. 12 Regional offices reviewed were the Northwest Mountain, Southern, Southwest, and Western-Pacific; Flight Standards district offices were Atlanta, Dallas, Dallas/Fort Worth, Fort Worth, Los Angeles, Miami, Scottsdale, and Seattle; offices with international responsibility were Brussels, Dallas/Fort Worth, Frankfurt, London, Miami, and San Francisco. Page 25 GAO/RCED-98-21 Aviation Safety Chapter 1 Background response rate of 90 percent, and its results can be generalized to all FAA inspectors with responsibility for repair stations. We conducted our review from August 1996 through October 1997 in accordance with generally accepted government auditing standards. In September 1997, we provided the Department of Transportation and FAA with a draft of this report for their review and comment. We met with FAA officials, including the Deputy Associate Administrator for Regulation and Certification (acting on behalf of the FAA Administrator) to obtain FAA’s comments. Those comments and our responses are included in the executive summary and chapters 2, 3, and 4. For a more detailed discussion of our scope and methodology, see appendix I. Page 26 GAO/RCED-98-21 Aviation Safety Chapter 2 Current Inspection Approach Limits FAA’s Ability to Ensure Compliance Although FAA is meeting its oversight goal to inspect every domestic and foreign repair station at least once a year, the use of one-person inspections at large, complex facilities restricts the agency’s ability to identify deficiencies and ensure compliance with regulations. We reviewed 19 instances in which FAA conducted a special team inspection of a facility that had received a one-person inspection within the previous year. These special team inspections identified far more deficiencies than inspections done by individual inspectors. Team inspections tend to be more comprehensive and focused, and team members are more organizationally independent of the repair station and have a more standardized approach to ensuring that all aspects of compliance with rules and regulations are checked. Many inspectors acknowledged the advantages of using a team rather than an individual inspector to review such facilities, stating that the pressure of other duties keeps them from conducting inspections on their own that thoroughly identify deficiencies and, thus, ensure compliance. Some FAA offices we visited have developed ways to conduct inspections using teams rather than individual inspectors and to do so without adversely affecting other demands on inspectors’ time. Their actions hold promise as a “best practice” that FAA could examine for broader application. Surveillance is one of the most important functions FAA inspectors perform Number of to ensure safety and regulatory compliance in the aviation system. Each Inspections year, FAA identifies specific surveillance activities that must be conducted Conducted Meets the during the year, including an inspection of each repair station. This inspection is conducted by the FAA Flight Standards district office that Required Minimum maintains a repair station’s certificate. According to FAA’s guidance, the inspection is to cover all aspects of a repair station’s operations, including the currency of technical data, facilities, calibration of special tooling and equipment, and inspection procedures. The inspection is also to ensure that the repair station is performing only work that it has approval to do. While FAA’s guidance does not prescribe precisely how each inspection must be conducted, it provides some direction on how to perform a repair station inspection. It does not require inspectors to follow checklists or other prescribed approaches to conduct the inspection. FAA’s guidance requires, at minimum, one maintenance or avionics facility inspection of each repair station per year. Those repair stations with both maintenance and avionics ratings receive at least two facility inspections, one examining maintenance capabilities and the other, avionics functions. Page 27 GAO/RCED-98-21 Aviation Safety Chapter 2 Current Inspection Approach Limits FAA’s Ability to Ensure Compliance The standard of one inspection per year has not changed in recent years as air carriers have increased their reliance on repair stations.1 All 2,800 repair stations in the United States and around the world doing work on aircraft flown by FAA-certified air carriers received the inspections FAA’s guidance required in fiscal year 1996, according to officials at FAA headquarters. As partial verification of the FAA officials’ statement, we reviewed FAA’s Program Tracking and Reporting Subsystem (PTRS) data from the 13 FAA offices we visited to determine if the offices had made the facility inspections of the repair stations assigned to them. In all, these 13 offices were responsible for more than 950 inspections at over 750 repair stations working for FAA-certified air carriers. Our analysis of the data confirmed that these minimum inspection requirements were met. Type of Inspection Varies How repair stations are inspected varies based on decisions made by both FAA managers and the inspectors themselves. The approach also varies depending on whether the repair station is in the United States or abroad. Moreover, review of some repair stations’ activities is not limited to the annual facility inspection. Each year, FAA selects a few facilities for special, in-depth inspections, which FAA officials stated complement the surveillance conducted by individual inspectors. In the past 4 years, an average of only 23 of these inspections have been conducted at repair stations per year (less than 1 percent of the repair stations performing work for air carriers). In practice, most facility inspections of domestic repair stations are conducted by the individual inspectors who have been assigned the oversight responsibility for the repair stations. This approach is FAA’s front line of surveillance of repair stations. The inspectors assigned responsibility for repair stations are also assigned oversight of other aviation activities such as air taxis, agricultural operators, helicopter operators, and training schools for pilots and mechanics. In addition, the inspectors have other duties such as certifying new operators and investigating accidents and incidents. In performing routine surveillance, an inspector may make repeated visits to a single facility to complete the inspection because there is too much to accomplish in just one visit. This is particularly true at larger, more 1 Under fiscal year 1997 guidelines, FAA required inspectors conducting facility inspections to also inspect the repair stations’ procedures for the detection of suspected unapproved parts. Page 28 GAO/RCED-98-21 Aviation Safety Chapter 2 Current Inspection Approach Limits FAA’s Ability to Ensure Compliance complex repair stations. Inspectors responsible for such repair stations told us that they often make multiple visits to complete a single inspection. FAA’s guidance to inspectors also recognizes that because the size of repair stations can vary from a one-person operation to a large overhaul facility, the size and complexity of the facility may warrant the inspection being conducted by a team, rather than by an individual inspector. Some FAA offices do, in fact, assign teams to inspect some facilities. Like domestic repair stations, foreign repair stations are inspected every year. Unlike domestic repair stations, however, foreign repair stations must renew their certification with FAA at least every 2 years. The renewal inspection assesses whether the foreign repair station continues to meet Federal Aviation Regulations and fulfills FAA’s requirement for an annual facility inspection. The renewal inspection and the facility inspection cover the same aspects of repair station operations, according to FAA officials and inspectors with both domestic and foreign oversight experience. Like the facility inspection, the renewal inspection can be performed by an individual or by a team of inspectors. In the six offices we visited with responsibility for the oversight of foreign repair stations, both types of inspections were generally conducted by teams, particularly at larger repair stations. Each year, FAA does special, in-depth inspections at a small portion of the repair stations in the United States and abroad through its National Aviation Safety Inspection Program (NASIP) or its Regional Aviation Inspection Program (RASIP). FAA determines which facilities should receive additional oversight through these comprehensive reviews, selecting them on the basis of submissions from district and regional offices. In general, inspectors recommend, through their offices, facilities for special inspections based on inspection results or other reasons such as the size and complexity of operations. Although FAA’s emphasis has been on in-depth inspections of air carriers, repair stations have been part of the special inspection effort. In fiscal years 1993 through 1996, FAA conducted 428 special, in-depth inspections, 92 (or 21 percent) of which were of repair stations. Unlike the facility or renewal inspections, special inspections are performed by teams of inspectors that are independent of the district offices that have oversight responsibility for the carriers or facilities being inspected. Page 29 GAO/RCED-98-21 Aviation Safety Chapter 2 Current Inspection Approach Limits FAA’s Ability to Ensure Compliance Individual inspectors generally identify far fewer deficiencies than teams Team Inspections do. Although most repair stations are not inspected by both individuals Provide More Detailed and teams, at the FAA offices we reviewed, 16 repair stations routinely Review Than Those inspected by individuals were also inspected by one or more special teams during fiscal years 1993 through 1996. These teams found a total of 347 Conducted by deficiencies, of which only 15 (or 4 percent) had been identified by the Individual Inspectors individual inspectors in the 12 to 18 months prior to the special facility inspections. Because many of the deficiencies relate to work on specific aircraft or components, and because aircraft or components at a repair station vary from day to day, some variation in inspection findings is to be expected. However, a close look at the results suggests that individual inspectors, even when they make multiple visits to repair stations, may not identify many of the deficiencies that teams find. The special inspections we reviewed turned up many systemic deficiencies, such as problems with training or quality assurance, that appeared to be long-standing and that therefore could have been detected in earlier inspections. For example, a team conducting a special inspection found that a repair station’s manual contained procedures for aircraft fuel servicing and fuel tank maintenance that may have been counter to the policies of the air carriers for which the work was done. The individual inspector, who had visited this repair station many times in the previous 18 months had not reported this problem. Often, the deficiencies identified in the special inspections but not in the regular inspections were significant. The findings of special inspections are categorized as (1) violations of Federal Aviation Regulations, (2) violations of the repair stations’ FAA-approved repair station manuals, or (3) lack of systems to ensure continuing compliance. Of the deficiencies reported in the 19 special inspections on 16 repair stations we reviewed, one-third involved violations of FAA regulations (see fig. 2.1).2 For example, an inspection team found that a repair station was not segregating new and serviceable parts from those parts that were not serviceable. In another case, a repair station on three occasions approved an aircraft for return to service following a major repair that, according to the inspector’s report, was not completed “based on FAA-approved technical data.” 2 FAA does not have comparable data for deficiencies identified during regular inspections. Deficiencies are categorized only for special inspections. Page 30 GAO/RCED-98-21 Aviation Safety Chapter 2 Current Inspection Approach Limits FAA’s Ability to Ensure Compliance Figure 2.1: Types of Deficiencies FAA Teams Identified During Special, In-Depth Inspections Violations of FAA regulations 36.6% 37.5% Violations of FAA-approved manuals 25.9% Lack of systems to ensure continued compliance Note: Percentages are based on a total of 317 reported deficiencies. Although 347 deficiencies were reported, 30 were not categorized. Source: FAA’s data. Violations of a repair station’s approved manual also accounted for about one-third of the deficiencies. For example, one team found a repair station did not inspect subcontracted work in accordance with its manual. A floor mechanic was performing these inspections, rather than the quality control inspector. We contacted 13 inspectors responsible for the repair stations covered by the 19 special inspection reports to obtain their views on why the special inspections found so many more deficiencies, including ones that appeared to be long-standing. They said the pressure of other duties kept their individual reviews from being more comprehensive. For example, one inspector was responsible for 7 other repair stations, 11 air taxi operators, 3 helicopter and agricultural operators, 11 executive aircraft, and more than 30 airmen. In addition, while they were at the repair stations, inspectors had to deal with employees’ questions or concerns about matters unrelated to the inspections. All 13 inspectors said that for reasons such as these, an individual inspector has a greater chance of not identifying deficiencies, even after repeated visits. Page 31 GAO/RCED-98-21 Aviation Safety Chapter 2 Current Inspection Approach Limits FAA’s Ability to Ensure Compliance Attributes of Quality The quality of repair station inspections is important because surveillance Inspections More is one of FAA’s primary means for ensuring that repair stations continually Prevalent in Team meet Federal Aviation Regulations. FAA’s guidance to inspectors states that if surveillance is to meet its intended purpose, quality inspections are Inspections Than in essential. We developed four characteristics of a quality inspection based Individual Inspections on our initial discussions with FAA officials and inspectors, staff from the U.S. Department of Transportation’s Office of the Inspector General who were involved with repair station work, Department of Defense officials responsible for audits of carriers with contracts for transporting military personnel, and airline quality assurance officials. These four characteristics, explained in table 2.1, are independence, comprehensiveness, focus, and standardization. In subsequent discussions, FAA officials and inspectors agreed that a quality inspection should have these four characteristics. Table 2.1: Characteristics of a Quality Inspection Characteristic Definition/explanation Independence Inspectors need to be free of undue interest or complacency regarding the repair station’s operation. Inspectors who must deal with a repair station on an ongoing basis can lose their objectivity because they may feel they already know that the repair station’s operations are in good order. Comprehensiveness Each inspection needs sufficient time to cover all of the elements that are supposed to be covered. Focus The inspection needs to be performed without distraction. This means minimizing competing demands, such as dealing with repair station employees’ questions or concerns about other matters. Too many distractions can prevent inspectors from conducting a thorough inspection. Standardization Even though repair stations are different, each one needs to be reviewed for all of the applicable requirements. Use of an agreed-upon checklist or job aid helps to ensure that all similar inspections are conducted in a similar fashion. The size and complexity of many large repair stations are such that an individual inspector may have difficulty maintaining these quality characteristics. In examining FAA’s facility inspection records, and in discussions with inspectors, we found these four characteristics were Page 32 GAO/RCED-98-21 Aviation Safety Chapter 2 Current Inspection Approach Limits FAA’s Ability to Ensure Compliance more prevalent in facility inspections conducted by teams than in those conducted by individual inspectors. Independence Inspectors assigned to teams have no ongoing responsibility for the repair station and have no relationship with its operator. By contrast, the individual inspector who conducts a facility inspection is usually the one assigned to manage the repair station’s certificate for an extended period of time. During this period, the primary contact the repair station has with the FAA is through this inspector. Inspectors we spoke with during field visits consistently stated that an inspector with a “fresh set of eyes” often identifies deficiencies that the principal inspector misses. Comprehensiveness Teams cover all subject areas during the course of their inspections, whereas individual inspectors’ other duties may limit the time they can spend and the extent of work they can do during their visits to repair stations. Many of the inspectors responding to our survey indicated that their ability to conduct a quality inspection was affected by factors related to comprehensiveness (see fig. 2.2). For example, 75 percent of the respondents said having too many inspection duties affected their ability to conduct comprehensive inspections to some degree, with 43 percent saying it was a major reason for the problem. (For a more detailed breakdown of the survey results for these and other survey topics, see app. III.) In addition, inspectors we spoke with at field offices said that it was very difficult to cover everything at a large or complex repair station. For example, one inspector said the week he spends at a large engine repair station is not enough time to complete a facility inspection. The size of the repair station and the complexity of the work being done, he said, makes it difficult to ensure that he is making a comprehensive inspection. Page 33 GAO/RCED-98-21 Aviation Safety Chapter 2 Current Inspection Approach Limits FAA’s Ability to Ensure Compliance Figure 2.2: FAA Inspectors’ Responses on Barriers to Comprehensive 50 Percentage of respondents Inspections 40 30 20 10 0 Too many Not enough Difficult to inspect inspection time to visit larger facilities duties to be repair stations because of comprehensive as often as size/complexity desired Major reason hindering success in ensuring compliance Minor reason Not a reason Note: Percentages may not add to 100 because a few respondents said they had no basis to make a judgment. Focus In a team inspection, completing a portion of an inspection is the only duty of each team member. An inspector conducting an inspection alone faces work demands from other locations as well as divided responsibilities at the repair station being inspected. Inspectors’ responses to our survey also reflected concerns about their ability to focus sufficiently during inspections (see fig. 2.3). For example, 80 percent of the inspectors responding indicated that spending time on other duties had an effect on the quality of the inspections they performed. For example, during our interviews, inspectors said they needed to spend time during inspection visits answering questions or clarifying regulations for repair station employees. They said such duties were part of their job, but some noted that these conflicting demands can interfere with their ability to focus on the inspection they are trying to conduct. Page 34 GAO/RCED-98-21 Aviation Safety Chapter 2 Current Inspection Approach Limits FAA’s Ability to Ensure Compliance Figure 2.3: FAA Inspectors’ Responses on Barriers to Focusing During Inspections 60 Percentage of respondents 40 20 0 Need to spend too Need to spend time much time on providing education duties other than and guidance to inspections operators Major reason hindering success in ensuring compliance Minor reason Not a reason Note: Percentages may not add to 100 because a few respondents said they had no basis to make a judgment. Standardization Inspection results can be more useful to inspectors and FAA if there is assurance that all areas have been adequately covered. If all areas are covered, inspectors have greater assurance that the repair station complies with regulations. Checklists or other similar job aids are one way to provide this assurance and to do so in a structured, consistent manner. A checklist or similar job aid for repair station inspections would include all areas that inspectors must review as part of the inspection as well as how the regulations governing repair station activities relate to these areas. At present, however, FAA does not require the use of a checklist during a repair station inspection. We found evidence from a number of sources that the use of an effective checklist plays an important role in a thorough inspection. Officials from FAA, industry, and the Department of Defense (which reviews air carriers before awarding defense contracts) told us that they would question the Page 35 GAO/RCED-98-21 Aviation Safety Chapter 2 Current Inspection Approach Limits FAA’s Ability to Ensure Compliance comprehensiveness of any facility inspection of a repair station that was not done using a job aid or checklist. Air carriers and the Department of Defense reported that their own inspectors use such aids to guide their work. They said the scope of the inspection of many repair stations is large enough that it is not difficult to overlook a portion of what must be covered. While FAA does not require the use of a checklist or job aid for routine surveillance, teams, whether conducting routine or special inspections, are more likely than individual inspectors to use checklists or other job aids that help ensure that all areas are covered, based on our observations. We found that during team inspections, team members use the same structured approach, typically in the form of an inspection job aid or checklist, such as the NASIP checklist. For example, one overseas office that conducts its inspections with in-house teams has a job aid covering each portion of the inspection. By contrast, we found that the approaches used by inspectors conducting their own inspections varied greatly, and individual inspectors were less likely to use checklists or other job aids to ensure that all areas had been covered. For example, while one inspector showed us a detailed checklist he developed combining guidance from the Airworthiness Inspector’s Handbook with the regulations applicable to repair stations, others said they do not use any job aid and work instead from memory when inspecting repair stations. We also found that when individual inspectors use a checklist, they tend to use one that is not detailed enough to ensure that compliance with regulations is checked. FAA’s standard, most commonly used job aid, which is available to inspectors through the Flight Standards Automated System, is not directly tied to the standards that repair stations must meet. Although it lists items to review, it does not provide references to the regulations governing repair stations. By contrast, the checklist used during NASIP inspections provides this link, as do the checklists and job aids the aviation industry uses to evaluate repair stations. Officials at the regional and office levels have indicated that knowing how an inspection finding relates to the regulations is important for pursuing enforcement action when a violation is identified. We found one office that requires all inspectors to use a job aid tied to the regulations. The office manager said that by having all inspectors use a standardized approach, he has greater assurance that effective and comprehensive inspections are being performed and that repair stations are in compliance with regulations. Page 36 GAO/RCED-98-21 Aviation Safety Chapter 2 Current Inspection Approach Limits FAA’s Ability to Ensure Compliance We asked FAA headquarters officials what they thought of encouraging inspectors to make greater use of checklists and other job aids. They said that guidance and job task lists provided to inspectors encourage the development of good work processes by each inspector without removing the flexibility required for them to evaluate a repair station’s compliance. However, the greater use of checklists or other job aids to help ensure that comprehensive inspections are being performed could be instituted in a way that does not diminish the inspectors’ flexibility. At a minimum, these types of tools would serve to remind the inspectors of the elements of the inspection that are the most critical to safety. Inspectors Describe Most inspectors responding to our survey responded favorably when Overall Repair Station asked for their general impressions about repair stations’ overall Compliance Favorably, but compliance with regulations. Sixteen percent of the inspectors put compliance at that top, or “excellent” level, 68 percent rated compliance as See a Need for “good,” and 12.5 percent rated it as “fair.” (See fig. 2.4.) Improvement in Many Areas Figure 2.4: FAA Inspectors’ Views on the Current Overall Compliance of Repair Stations Excellent 68% Good 16% 12.5% Fair Poor/Very Poor 2% Uncertain 1.5% Page 37 GAO/RCED-98-21 Aviation Safety Chapter 2 Current Inspection Approach Limits FAA’s Ability to Ensure Compliance Inspectors acknowledged, however, that there was room for improvement. We asked inspectors about eight areas of compliance, such as the repair stations’ use of up-to-date manuals from manufacturers and air carriers. In each of the eight areas, more than half of the inspectors surveyed saw the need for at least some improvement. Specifically, in none of the eight categories was the percentage higher than 38 percent for respondents who thought little or no improvement was needed. By contrast, the percentage of inspectors who saw a need for some or moderate improvement ranged from 39 to 52 percent, and those who saw the need for great or very great improvement ranged from 18.5 to 33 percent. (See fig. 2.5.) Page 38 GAO/RCED-98-21 Aviation Safety Chapter 2 Current Inspection Approach Limits FAA’s Ability to Ensure Compliance Figure 2.5: FAA Inspectors’ Responses on Amount of Improvement Needed for Repair Station Compliance 60 Percentage of respondents 50 40 30 20 10 0 Repair Repair Actual Altered/ Require- Air carrier's Repair Supervisors at station's station is procedures of repaired ments met for current station's repair stations inspection using repair station parts equal in recording manuals inspection receive training procedures manufac- give condition to mainte- and program from air manual gives turers' satisfactory original parts nance procedures complies with carriers satisfactory current control are being air carrier's quality control manuals used maintenance program Great/very great improvement needed Some/moderate improvement needed Little/no improvement needed Note: Percentages may not add to 100 because a few respondents said they had no basis to make a judgment. How could FAA offices, already facing a diverse and extensive mix of Different Approaches responsibilities, do a better job of inspecting repair stations without to Repair Station adversely affecting other operations? To determine if there were workable Surveillance Show answers to that question, we turned to the field offices themselves. FAA field offices are given some flexibility by FAA headquarters in deciding how Promise Page 39 GAO/RCED-98-21 Aviation Safety Chapter 2 Current Inspection Approach Limits FAA’s Ability to Ensure Compliance to accomplish their surveillance programs. We looked to see if some of these offices had developed alternative approaches that might hold promise for other locations. We identified several offices that had adopted approaches that might prove useful on a broader scale. In general, their practices fell into two main categories: (1) placing greater emphasis on the oversight of repair stations and (2) finding ways to shift local staff resources so that they could conduct more repair station inspections with teams rather than with individual inspectors.3 FAA headquarters is also examining a revised approach to surveillance that could help improve the inspection process. Placing Higher Priority on Some of the FAA field offices we reviewed, departing from the standard Inspections of Repair approach to assigning inspectors’ responsibilities, have developed new Stations approaches on their own to place greater emphasis on repair stations. Typically, inspectors working in a field office are separated into two disciplines: general aviation and air carrier. General aviation inspectors are assigned to specific repair stations and also inspect operators covered by part 135 of the Federal Aviation Regulations—that is, air taxi operators. In addition, they inspect other aspects of the industry such as agricultural aircraft operators, technical schools for pilots and mechanics, and helicopter operators. Air carrier inspectors are responsible for operations covered by part 121 of the Federal Aviation Regulations—that is, for domestic air carriers. They are assigned specific carriers to inspect, and may, in that context, inspect the carriers’ in-house repair stations or those the carriers use, assessing whether repairs being made conform with the carriers’ FAA-approved maintenance manuals. Two FAA offices we visited have found this structure does not recognize the importance of overseeing repair stations and are pursuing other approaches that place a higher priority on it. Officials at FAA headquarters said they supported these efforts but will first evaluate the results and then, if appropriate, use the approaches at other offices. Scottsdale Flight Standards In the Western-Pacific Region, officials have approved a new District Office, Western-Pacific organizational structure at the Flight Standards district office in Region Scottsdale, Arizona. Under the new structure, the district office is divided into an air transport and an air commerce unit. The air transport unit oversees air carriers and large repair stations, while the air commerce unit 3 One criterion for our consideration of possible approaches was that the approach require no additional resources beyond what the field office believed it could commit to the oversight of repair stations at current or anticipated staffing levels. However, under staffing increases approved by the Congress, many field offices will have more inspectors. Ch. 4 discusses the staff increases in greater detail. Page 40 GAO/RCED-98-21 Aviation Safety Chapter 2 Current Inspection Approach Limits FAA’s Ability to Ensure Compliance oversees air taxis, technical schools for pilots and mechanics, and other operators. Within the air transport unit, one team focuses its work on the five largest repair stations in the area, all of which perform heavy airframe maintenance. According to a district office supervisor, two additional staff members with significant repair station experience have been hired in the last 3 years. The expanded staff made it possible to establish a repair station team without asking for additional resources. Large-component repair stations, such as those working on landing gear or engine parts, may be added to the team’s responsibility at a future date, an official said. According to district office staff, the office has been concerned with FAA’s lack of surveillance of larger, more complex repair stations. Inspectors stated that the reorganization will allow them to spend more time on those facilities without affecting the surveillance of smaller repair stations. In conjunction with the reorganization, office management also redistributed the repair station workload among inspectors to allow them to provide more effective surveillance of larger facilities. Seattle Flight Standards District The Seattle Flight Standards district office revised the position description Office, Northwest Mountain of several inspectors to place a greater emphasis on the oversight of repair Region stations. Under these revised position descriptions, three maintenance inspectors, each with an assistant, will be responsible for the nine largest and most complex repair stations in the district. According to a district office official, FAA headquarters must approve this change because under the present set of position descriptions for inspectors, all repair stations are considered to have the same degree of complexity. In a letter to the regional office justifying the new positions, the district office manager stated that the office has not had the resources to “become proactive in the day-to-day activities of the facilities.” For example, at a repair station that works on over 400 aircraft annually with a staff of 2,000, adequate surveillance was not provided in several areas, such as compliance with customer airline procedures and regulatory requirements, according to office staff. In addition, adequate spot checks of maintenance performed by the company had not been made. The inspectors responsible for this facility were responsible for other facilities as well and did not have enough time to do a comprehensive review of the repair station. Under the new position descriptions, the principal inspectors have fewer responsibilities and so will be able to spend more time at each of the large facilities assigned to them. As in Scottsdale, the workload for the inspectors in Seattle will change dramatically. For example, one inspector will go from overseeing 16 repair stations, 8 air taxi operators, 4 executive Page 41 GAO/RCED-98-21 Aviation Safety Chapter 2 Current Inspection Approach Limits FAA’s Ability to Ensure Compliance aircraft operators, 2 helicopter operators, 2 agricultural operators, 45 airmen, and a pilot school to overseeing just 7 repair stations, all of which are facilities working on aircraft component parts. The remaining responsibilities will be distributed among existing and projected additional staff.4 European International Field The Eastern Region has four international field offices, three in Europe Offices, Eastern Region and one in New York.5 Because the European offices are not faced with many of the other responsibilities that domestic offices must handle—such as overseeing pilot and mechanic schools, agricultural aircraft, and certificate management of air carriers—the primary focus of their work program is on the certification and surveillance of repair stations along with limited surveillance of U.S. air carriers. (The New York office handles other responsibilities such as the oversight of foreign air carriers and the International Aviation Assessment Program.) Consequently, inspectors at the European offices are able to spend more time on the surveillance of repair stations than their U.S.-based counterparts. All of the inspectors we interviewed at the European offices said they spent 80 percent or more of their time on repair stations, whereas inspectors at domestic offices said they spent only about 30 percent of their time on surveillance of all types of facilities, including repair stations. Conducting More Locally A second and closely related development we observed was the use of Based Team Inspections locally based teams to conduct surveillance. The use of in-house teams in these offices ranges from making them the typical surveillance approach, as in the international field offices, to using them occasionally for areas in which problems have been identified. Because these teams are made up of local office staff, the cost is lower than for special inspections conducted by NASIP or RASIP teams assembled from around the country or around the region. Moreover, because local resources are used, the office can assess the effect of this approach on the office’s other responsibilities. The move toward team-based surveillance inspections was supported by the inspectors we surveyed. Officials at FAA headquarters also said they support these efforts but will assess them before asking other offices to make greater use of in-house teams. 4 See ch. 4 for further discussion of FAA’s approved staff increases. 5 The three European offices are in London, Brussels, and Frankfurt. Page 42 GAO/RCED-98-21 Aviation Safety Chapter 2 Current Inspection Approach Limits FAA’s Ability to Ensure Compliance Scottsdale Flight Standards Prior to the office’s restructuring, staff at the Scottsdale Flight Standards District Office, Western-Pacific district office routinely performed team inspections on the largest repair Region stations each year. They also performed team inspections on selected smaller facilities. According to one official at the office, team inspections are a big part of the overall surveillance program because the office believes such inspections are the only way it can ensure that the repair stations are meeting all applicable regulations. Use of team inspections is expected to increase under the new office organization. Team inspections at the Scottsdale office are led by the principal inspector, the person with the most knowledge about a repair station. Seattle Flight Standards District The Seattle Flight Standards district office has increasingly relied on Office, Northwest Mountain in-depth reviews conducted by teams of inspectors as a way to strengthen Region its oversight of repair stations. According to officials in the region’s Flight Standards Division, current surveillance of repair stations, as well as surveillance of other certificate holders, is not as effective as it should be. As evidence, they cite national statistics indicating that only five enforcement actions (such as a warning notice or a civil penalty) result from every 1,000 inspections FAA conducts (an enforcement rate of 0.5 percent). By contrast, NASIP inspections, which are more in-depth, result in an enforcement rate of 20 percent. District office officials said that the team approach is being used so that the staff is more aware of what is happening at the facilities they oversee. Moreover, the office has found team inspections conducted to date to be very successful. For example, a recent team inspection at a large component shop repair station identified 17 deficiencies that the principal inspector said he had not identified in several prior inspections. According to the inspector, his workload and the complexity of this repair station prevented him from performing an inspection comprehensive enough to identify the kinds of deficiencies found by a team. This inspector, along with others we interviewed at the district office, agreed that team inspections are necessary for adequate surveillance in some cases, particularly for larger, more complex repair stations. Miami Flight Standards District The Miami Flight Standards district office has established a quality Office, Southern Region assurance unit that, among other activities, performs team inspections of repair stations. The teams inspect air carriers, repair stations, and other operators in response to complaints or an inspector’s request. According to a regional office official and the district office supervisor of the inspection teams, the inspections are more objective and comprehensive than the routine inspections. In addition, the inspections allow the team to Page 43 GAO/RCED-98-21 Aviation Safety Chapter 2 Current Inspection Approach Limits FAA’s Ability to Ensure Compliance identify and correct potential problems that if left unaddressed could develop into compliance problems. From fiscal year 1993 through 1996, 32 in-depth team inspections were conducted by the office, 14 of them at repair stations. International Field Offices At all six of the offices we visited that oversee foreign repair stations, inspections are typically conducted by teams, although smaller repair stations with very few employees or capabilities may be assigned to only one inspector. For example, the Frankfurt office specifies that annual surveillance on a repair station with more than 100 employees be performed by a team of up to five inspectors. The team approach is used because the office does not believe an individual inspector can cover an entire facility. According to the office’s manager, a team provides broader, deeper coverage and the end result is that more deficiencies are identified. As in the NASIP and RASIP special inspections, segments of the facility inspection are divided among the participating inspectors. For example, one inspector will review the landing gear and window repair shops, while another will inspect the technical library and the calibration laboratory. The principal inspector assigned to the facility acts as the team leader and prepares the findings presented to the repair station at the conclusion of the inspection. Inspectors responsible for foreign repair stations said that although they may visit some repair stations only once a year, less than when they performed domestic repair station surveillance, the surveillance of foreign repair stations is more thorough because of the team approach. Inspectors Support Greater Use A substantial number of the inspectors we surveyed supported the use of of Team Inspections more team inspections. Figure 2.6 shows survey responses on using team inspections to improve compliance. Results show that 71 percent of the inspectors responding favor using team inspections staffed from within the district office, and 50 percent favor an increase in NASIP or RASIP inspections staffed from offices nationwide or within the region. Opposition was weak both to locally based team inspections (11 percent) and to the increased use of NASIP or RASIP inspections (16 percent). Page 44 GAO/RCED-98-21 Aviation Safety Chapter 2 Current Inspection Approach Limits FAA’s Ability to Ensure Compliance Figure 2.6: FAA Inspectors’ Responses on Using Teams to Improve Compliance 80 Percentage of respondents 60 40 20 0 Encourage team Increase inspections within regional and the individual office national team inspections In favor Neutral Opposed Note: Percentages may not add to 100 because a few respondents said they had no basis to make a judgment. This support was affirmed in our interviews with inspectors at the offices we visited. One inspector stated that while he worked in the airline industry, the company would never send fewer than two inspectors to a contract repair station. Still other inspectors stated that individually, they are unable to obtain reasonable assurance of compliance with regulatory requirements at larger facilities. New Inspection Approach Other developments within FAA may have future implications for how for Air Carriers May Have repair station inspections are conducted. One recommendation from FAA’s Applicability for Repair 90-Day Aviation Safety Review completed in September 1996 was the creation of the Certification Program Office, which would include a Stations National Certification Team to assist local Flight Standards district offices in processing new air carrier certifications. In addition, the new office will Page 45 GAO/RCED-98-21 Aviation Safety Chapter 2 Current Inspection Approach Limits FAA’s Ability to Ensure Compliance also include a centralized safety analysis and information management office that will assist inspectors in targeting surveillance resources and taking necessary corrective actions to mitigate safety risks. These approaches to improving the surveillance of air carriers can also be applied to the surveillance of repair stations’ operations. In a separate effort, FAA is testing a method of surveillance that emphasizes the compliance with specific regulations rather than the completion of a series of inspections. According to an FAA official involved with this test, FAA is examining this new approach because it is concerned that the current approach does not adequately link inspections to specific regulations. The test is being done on air carriers, not repair stations, but it could potentially be extended to repair stations, according to FAA personnel. FAA’s current inspection approach is based on the National Program Work Guidelines, issued annually by FAA headquarters. These guidelines list specific inspections that must be completed. The guidance tells inspectors what types of inspections to perform, but it does not tell them what regulations they are to verify compliance with. The new approach, called “virtual recertification,” works in much the opposite way. Instead of specifying the types of inspections to perform, it specifies the applicable safety regulations to be checked and leaves it up to the inspector to determine how to verify compliance. Given this emphasis, the inspector must ensure that surveillance activities are comprehensive enough to cover all aspects of the regulations. For example, an inspector would verify that the repair station is meeting requirements under 145.47(b), specifically, that the repair station ensures that all inspection and test equipment is tested at regular intervals to ensure correct calibration to a standard derived from the National Bureau of Standards. This approach may prove more successful at ensuring that important safety requirements are not omitted from surveillance. FAA is testing the approach in one region, where it is being applied to two air carriers. The region would like to extend the test to one large repair station as well, according to FAA officials. Although it is too early to judge the effects of this test on FAA’s approach to surveillance and its potential effect on repair stations is unknown, it may influence how FAA headquarters and offices adjust their future oversight of repair stations to provide the maximum benefit with the limited resources available. Page 46 GAO/RCED-98-21 Aviation Safety Chapter 2 Current Inspection Approach Limits FAA’s Ability to Ensure Compliance FAA appears to have the opportunity to enhance the effectiveness of its Conclusions repair station inspections. While just one inspector may be sufficient to conduct surveillance on smaller or more specialized repair stations, this approach does not appear to be nearly as effective at large, complex facilities. At such facilities, team inspections have proven more effective in identifying deficiencies. In addition, team inspections do a better job of incorporating the four characteristics of quality inspections, in that they are more independent, comprehensive, focused, and standardized than inspections conducted by individual inspectors. Acting on their own, several FAA offices are reconfiguring their staffs and adjusting their operations to conduct more team inspections. Their approaches hold promise both for making more efficient use of inspection staff and for improving the quality of surveillance. FAA headquarters officials support these efforts but will evaluate them before asking other offices to examine such an approach. We think it is appropriate for all district offices, especially those with high concentrations of repair stations, to reevaluate their organization and surveillance approach to determine if they can make better use of their current inspection resources. For example, an office may determine that local team inspections are appropriate and a good use of resources for repair stations that are large, complex, or have higher rates of noncompliance. An analysis of the widely varying inspection approaches also highlights the importance of a standardized checklist or other effective job aid in ensuring that inspections are comprehensive. Repair stations can be very complex, and a checklist can help ensure that all applicable areas are covered and that this coverage is consistent from facility to facility. Such checklists are in widespread use by other organizations, such as air carriers and the Department of Defense, that conduct similar types of inspections. Where they are already in use within FAA, they appear to improve both comprehensiveness and standardization. Wider use of such checklists appears to be another appropriate way to increase the effectiveness of FAA’s inspection effort. We recommend that the Secretary of Transportation instruct the Recommendations Administrator of FAA to (1) expand the use of locally based teams to conduct routine facility inspections, particularly for facilities that are large, complex, have higher rates of noncompliance, or meet predetermined risk indicators; and (2) develop and use checklists or job aids for inspectors that allow a greater degree of comprehensiveness, Page 47 GAO/RCED-98-21 Aviation Safety Chapter 2 Current Inspection Approach Limits FAA’s Ability to Ensure Compliance standardization, and assurance that the repair station complies with regulatory requirements. FAA agreed with the recommendations. With regard to expanding the use Agency Comments of locally based teams to conduct inspections, FAA headquarters officials said they support field office efforts currently under way but will assess these efforts before asking other offices to make greater use of in-house teams. While agreeing with the recommendation to provide better job aids for inspectors, FAA officials did not provide specific details on how or when they would implement this recommendation. In addition, FAA cited several agency initiatives that it said are under way to enhance its oversight of the repair station sector of the aviation industry. FAA said its 90-day safety review conducted last year recommended the creation of an analytical unit that could provide safety trend data to inspectors. FAA said an office within the Flight Standards Service was created on May 20, 1997, to provide data that will help focus inspection and other resources. The review also recommended that field and division managers be given flexibility to determine the skills needed in a particular field office to ensure the appropriate mix of technical, paratechnical, support, and clerical expertise. FAA said that this flexibility will be supported through the establishment of new staffing standards—a long-term project that is already under way. FAA also said that it had recognized the need to evaluate the air operators safety systems, including those of repair stations, and had initiated a Surveillance Improvement Program. Under this program, a team of safety inspectors, technical personnel, and managers, aided by Sandia National Laboratories, investigated ways to improve the surveillance process. The team recommended improvements in standardization and communication, as well as other areas that will allow FAA to evaluate compliance more effectively. Efforts are under way to implement the fundamental changes to surveillance that were recommended by this team. Page 48 GAO/RCED-98-21 Aviation Safety Chapter 3 Incomplete Documentation of Inspections Makes Extent of Follow-Up Difficult to Determine In many instances, we were unable to determine how well FAA was following up to ensure that repair stations corrected deficiencies identified during inspections. Particularly for domestic repair stations, the lack of documentation made it impossible to assess how quickly or thoroughly repair stations brought themselves into compliance. Documentation was better for foreign repair stations, which generally appeared to be correcting deficiencies quickly to qualify for renewal of their certificates. Resolving problems with documentation is particularly important because FAA is taking new steps to use its management information systems to determine where inspection resources should be targeted. Incomplete data can make such efforts less effective. Much of the value of inspection activity is not in finding and listing Information on problems but in resolving the problems effectively, according to FAA field Inspection Results Is office managers and supervisors. They said that although much of the Important but resolution may hinge on the working relationship between the FAA inspector or inspectors and the repair station’s personnel, effective Incomplete documentation of the actions taken is a necessary part of demonstrating what problems were found, what was done to resolve them, and whether all parties are in agreement that deficiencies have been corrected. FAA officials acknowledged that effective documentation of inspection and follow-up activity is needed. Types of Information FAA’s guidance is limited in specifying for inspectors what documents to Needed for Basic include in repair station files. FAA’s files on repair stations are the agency’s Documentation and official record of inspection-related activity—and therefore the backbone of any system that uses management information to help spot trends, Management Reports identify problems, and target inspection resources, according to field office managers and supervisors. The guidance points out generally that the kinds of documentation of inspections and surveillance activities include inspection reports and related correspondence, but the guidance does not specifically require that any document be included. The closest thing to a requirement is a statement in the Airworthiness Inspector’s Handbook that the letter to the repair station describing all deficiencies should be included in the case file. After analyzing FAA’s inspection and follow-up program, we determined that, at a minimum, the files need to contain the following if the extent to which repair stations are correcting problems in a timely manner is to be monitored: Page 49 GAO/RCED-98-21 Aviation Safety Chapter 3 Incomplete Documentation of Inspections Makes Extent of Follow-Up Difficult to Determine • a memo to the file or other documentation showing that an inspection was performed, what was inspected, and the results; • a deficiency letter from FAA informing the repair station of the problems that needed to be corrected; • a response from the repair station indicating what actions it was taking to address the deficiencies; and • a memo to the file or other acknowledgment that the repair station’s actions were an acceptable response and that the deficiencies had been resolved. FAA officials agreed that these items are important in developing complete supporting information about the extent to which deficiencies were being resolved in a timely fashion. We believe—and FAA officials agreed—that beyond effective documentation in the repair station files, FAA also needs an effective management information system for capturing this basic information, combining it with information from other activities, and synthesizing it in ways that allow management to plan surveillance activities, schedule manpower resources, evaluate accomplishments, analyze results for patterns or trends, and modify planned activities. FAA’s management information tool for its inspection activity is its Program Tracking and Reporting Subsystem (PTRS). To provide data for planning and oversight of FAA’s inspection program, inspectors record inspection results in the computer-based PTRS. FAA’s PTRS Procedures Manual requires that inspectors record comprehensive reports demonstrating that inspections were performed, including inspection results, whether the repair station took any action, and whether the inspector took any follow-up action to ensure that deficiencies were corrected. Our analysis of the system showed that the key items of information needed for useful management reporting are the following: • an indication that a repair station was inspected and the results; • an indication that all deficiencies were communicated to the repair station in a deficiency letter; and • an indication that the deficiency letter was “closed out” when corrective actions by the repair station were determined to be acceptable by the inspector. FAA officials agreed that PTRS should contain these items of information if the system’s reports are to be of substantial use. Page 50 GAO/RCED-98-21 Aviation Safety Chapter 3 Incomplete Documentation of Inspections Makes Extent of Follow-Up Difficult to Determine Extent of Documentation We did not find sufficient information in FAA’s repair station files to in Repair Station Files assemble a clear picture of how quickly and completely deficiencies found during the inspections of repair stations were being corrected. This was particularly true for domestic repair stations. In all, we reviewed the files on 331 domestic and 157 foreign repair stations for fiscal years 1993 through 1996.1 Determining the speed and completeness with which deficiencies were corrected was not possible for the following reasons: • When there was evidence that problems had been found, evidence of corrective action was usually absent. Of the 331 domestic files reviewed, 96 contained deficiency letters, indicating that the facility had been notified that problems existed. Response letters from repair stations were present for 73 percent of the deficiency letters. However, only 22 percent of the deficiency letters were accompanied by documentation showing that the repair stations’ responses about resolving deficiencies were acceptable (see fig. 3.1). • When there was no evidence in a file of a problem at a facility, this alone could not be taken as assurance that no problems had been found. Even for the 235 files that did not contain deficiency letters, it was not possible to assume that FAA inspectors had not identified deficiencies because the files contained no documentation showing that an inspection had been completed.2 1 Our set of 488 files was based on a judgmental sample taken at 11 of the 13 domestic and foreign FAA offices we visited. The sample ranged between 44 and 100 percent of those repair stations for which the office had oversight responsibility and was limited to only repair stations doing work for air carriers. 2 In our review of the files at domestic and foreign offices, when FAA’s documentation did not include copies of deficiency letters, we did not follow up to determine whether repair stations had actually received letters. Page 51 GAO/RCED-98-21 Aviation Safety Chapter 3 Incomplete Documentation of Inspections Makes Extent of Follow-Up Difficult to Determine Figure 3.1: Extent of Follow-Up Documentation for Inspections of Selected Domestic Repair Stations, Fiscal Years 1993 Through 1996 Was a deficiency letter present? If a deficiency letter was present, was documentation of subsequent follow-up in place? Deficiency letter Repair station's 96 repair stations response to deficiency letter FAA's analysis of repair station's response 0 25 50 75 100 b Percentage with documentation in place No deficiency letter 235 repair stations a a Files did not indicate whether these repair stations received satisfactory inspections. b Percentages are based on follow-up documentation for 172 deficiency letters (some repair stations had more than one deficiency letter during the period examined). Categories are exclusive of each other and do not add to 100. Our review of files on foreign repair stations found more complete documentation that follow-up had occurred (see fig. 3.2). We reviewed 157 files and found deficiency letters in 135. Response letters from repair stations were present for nearly 80 percent of the 356 deficiency letters in these 135 files. An even higher percentage—85 percent—contained some form of documentation indicating that FAA had followed up. For most repair stations, this documentation took a form not found in domestic files—a certificate renewal letter. Unlike domestic repair stations, foreign repair station certificates are subject to renewal by FAA at least every 24 months. In practice, many are renewed every year, according to FAA personnel. As part of the renewal, the FAA office issues a new certificate once it is satisfied that the repair station has taken appropriate actions to resolve the deficiencies. While certificate renewal letters were the most common form of follow-up documentation, nearly one-fourth of the files Page 52 GAO/RCED-98-21 Aviation Safety Chapter 3 Incomplete Documentation of Inspections Makes Extent of Follow-Up Difficult to Determine with deficiency letters also contained additional evidence of FAA’s analysis of the repair stations’ responses. Figure 3.2: Extent of Follow-Up Documentation for Selected Foreign Repair Station Inspections, in Fiscal Years 1993 Through 1996 Was a deficiency letter present? If a deficiency letter was present, was documentation of subsequent follow-up in place? Deficiency letter Repair station's 135 repair stations response to deficiency letter Recertification letter and/or other FAA analysis of repair station's response 0 25 50 75 100 b Percentage with documentation in place No deficiency letter a 22 repair stations a Files did not indicate whether these repair stations received satisfactory inspections. b Percentages are based on follow-up documentation for 356 deficiency letters (some repair stations had more than one deficiency letter during the period examined). Categories are exclusive of each other and do not add to 100. One office has taken a further step to tie the renewal letter more closely to the resolution of deficiencies. According to the manager of the Frankfurt field office, the office recognized in 1996 that files contained no formal documentation that a repair station had taken corrective action because the renewal letters did not specifically mention it. As a result, the office revised the renewal letter, adding a statement that reads, “We are pleased to inform you that the corrective actions and corrective action plan developed subsequent to the Frankfurt International Field Office repair station inspection, has been reviewed and accepted by the principal inspector(s).” The office began using these letters in August 1996. Page 53 GAO/RCED-98-21 Aviation Safety Chapter 3 Incomplete Documentation of Inspections Makes Extent of Follow-Up Difficult to Determine Extent of Documentation Documentation in PTRS is even less complete than documentation in the in Database Records individual files on repair stations. We examined PTRS records to determine the degree to which they contained inspection and follow-up information for the deficiency letters we found in our review of domestic and foreign repair station files. PTRS generally showed that an inspection had been conducted, but responses on actions to correct deficiencies were less frequently recorded than in the files, as were indications that closure had been reached. A great deal of inconsistency was reflected in the data entered into the system. Some inspectors entered inspection results in great detail, others entered only partial data, and still others entered no data at all about the deficiencies found. For example, the PTRS entry for one inspection said, “Discrepancies listed in letter dated 04/16/96,” and provided no indication whether deficiencies had been corrected. In another example, the deficiency letter showed the inspector had documented nine deficiencies, but PTRS showed only three of these deficiencies and did not show if any had been corrected. Lack of Reliable As far back as 1987, we have reported on FAA’s shortcomings in having Documentation Is a current and reliable information on key program elements. In 1991, we Long-Standing Problem reported that the data in PTRS were unreliable for providing information on the performance of FAA’s inspection program and were inadequate for ensuring the accomplishment of key elements of the inspection program.3 In 1995, we concluded that FAA may be building its future information management system from a number of databases that contain incomplete, inconsistent, and inaccurate data.4 Again in 1996, we reported that until FAA implements a strategy to improve the quality of its data, problems with data quality may limit the usefulness of the system and prevent FAA from realizing its full potential for targeting limited inspection resources to higher-risk activities.5 In response to our reports, FAA has developed and implemented a comprehensive strategy to improve data quality. The Department of Transportation’s Inspector General has made similar observations. In March 1994, the Inspector General reported that FAA inspectors do not routinely document items inspected at repair stations or 3 Aviation Safety: Problems Persist in FAA’s Inspection Program (GAO/RCED-92-14, Nov. 20, 1991). 4 Aviation Safety: Data Problems Threaten FAA Strides on Safety Analysis System (GAO/AIMD-95-27, Feb. 8, 1995). 5 Aviation Safety: New Airlines Illustrate Long-Standing Problems in FAA’s Inspection Program (GAO/RCED-97-2, Oct. 17, 1996). Page 54 GAO/RCED-98-21 Aviation Safety Chapter 3 Incomplete Documentation of Inspections Makes Extent of Follow-Up Difficult to Determine follow-up actions taken.6 In 1995, the Inspector General found that FAA inspectors were not interpreting PTRS reporting procedures consistently, resulting in inaccurate, inconsistent reports.7 Quality of Data Threatens The quality of PTRS data is important because PTRS is expected to provide Effectiveness of New data for FAA’s new information management initiative, the Safety Management Information Performance Analysis System (SPAS). SPAS is a computer-based analysis system designed to assist FAA in applying its limited inspection resources Initiative to those entities and areas that pose the greatest risk to aviation safety. This system, estimated to cost $32 million to develop and install, is also expected to highlight particular types of repair stations for increased surveillance or oversight because they are experiencing problems at rates that exceed the averages for that group. However, if the data on which SPAS is based are not complete and accurate, FAA could be limited in its ability to identify trends and target inspection resources. In the past, we have recommended data improvements as a preliminary step to implementing SPAS. Our 1995 report, which concluded that SPAS will not be effective if the quality of its data is not improved, recommended that FAA develop and implement a comprehensive strategy to make such improvements. FAA agreed with this recommendation. Although FAA initially intended to have its approach in place by the end of 1995, it was October 1996 when FAA issued a strategy that provides clear and measurable objectives for data quality, accurate assessments of the quality of the current data in each database (including an analysis and possible redirection of FAA’s existing initiatives to improve data quality), milestones for attaining the stated quality objectives, and estimates of the resources required. According to headquarters officials, full deployment of SPAS will be completed in December 1999, as required by legislation. Even so, until FAA completes the implementation of its strategy to improve data quality, problems with data quality will limit SPAS’ usefulness and prevent it from realizing its full potential. 6 Audit of the Certification and Surveillance of Domestic and Foreign Repair Stations: Federal Aviation Administration, Office of Inspector General, U.S. Department of Transportation, R4-FA-4-009 (Mar. 7, 1994). 7 Surveillance of Pilot Schools: Federal Aviation Administration, Office of Inspector General, U.S. Department of Transportation, R9-FA-002 (Nov. 8, 1995). Page 55 GAO/RCED-98-21 Aviation Safety Chapter 3 Incomplete Documentation of Inspections Makes Extent of Follow-Up Difficult to Determine While the lack of good documentation precludes a precise comparison of Compliance May FAA’s follow-up of deficiencies at foreign and domestic repair stations, Come More Quickly at some inspectors said that compliance comes more quickly at foreign Foreign Repair repair stations. We interviewed 34 FAA inspectors who had conducted inspections of both foreign and domestic repair stations, and they were Stations unanimous in concluding that compliance came more quickly at foreign stations. They said quicker compliance meant inspectors spent less time on follow-up and had more time for other work. The inspectors attributed this quicker compliance to the renewal requirement for foreign repair stations. Under FAA regulations, foreign repair stations must renew their certificates within 12 months of initial issuance and then at least every 24 months thereafter. By comparison, domestic repair stations retain their certificates indefinitely unless their operations are so badly run that FAA elects to take legal action to suspend or revoke their certificates. According to some inspectors, because new certificates cannot be issued until problems are resolved, foreign repair stations have an incentive to correct problems more quickly. Because of the poor documentation of the inspection results for domestic repair stations, however, we were unable to validate the inspectors’ views or to verify whether foreign repair stations achieve compliance in a more timely fashion than domestic repair stations. Opinions Vary on Within FAA, the strongest support for extending the certificate renewal Advisability of Extending requirement to domestic repair stations comes from inspectors who have Renewal Requirement to inspected both foreign and domestic repair stations. They solidly supported extending renewal to domestic repair stations to gain quicker Domestic Repair Stations resolution of noncompliance issues. Of the inspectors responding to our mail survey who had experience with foreign repair stations in fiscal year 1996, 89 percent supported extending the requirement. Similarly, 31 of the 34 inspectors we interviewed who had inspected both kinds of repair stations said they favored a domestic renewal requirement. In our discussions with inspectors, we were told that the lack of a renewal requirement makes inspectors less efficient because they must spend more time following up on repair stations with deficiencies and less time on much needed surveillance or other responsibilities. A specific deadline for correcting deficiencies to retain certification creates an incentive or sense of urgency to resolve deficiencies in a timely manner. Inspectors with experience only with domestic repair stations were somewhat less favorable toward certificate renewal. Our survey results show that 48 percent of the respondents who had experience only with Page 56 GAO/RCED-98-21 Aviation Safety Chapter 3 Incomplete Documentation of Inspections Makes Extent of Follow-Up Difficult to Determine domestic repair stations in fiscal year 1996 were not in favor of renewing domestic repair station certificates on a regular basis (for example, every 24 months) as a way to improve compliance. Our follow-up discussions with 39 inspectors who had experience only with domestic repair stations identified their reason for opposing such a requirement was a concern that substantially more work would be generated. However, this perception appears to be unfounded. The 34 inspectors we interviewed who had inspected both kinds of repair stations indicated that extending the requirement would not place additional requirements on inspectors. These inspectors said inspection requirements for renewal were the same as those for the annual facility inspection. Support for extending the renewal requirement was lowest among FAA management. They did not agree with inspectors who said that renewal would improve safety by obtaining quicker resolution of noncompliance problems. The Acting Manager of the Aircraft Maintenance Division said, for example, “No one has demonstrated that FAA would get one added ounce of safety if it revised the rule to require recertification of domestic repair stations.” They also raised several concerns about adopting this approach. Additional resources would be needed. According to the Acting Manager of the Aircraft Maintenance Division, it would take too many resources (staff time, etc.) to recertify every repair station. We pointed out that according to inspectors who had done both facility and renewal inspections, the requirements were the same. For the most part, he did not provide us with further information about what additional resources would be needed if the recertification requirement were extended. FAA attorneys, however, said they would need additional resources to process any cases stemming from the denial of recertification. Current procedures already allow certificates to be revoked. The current enforcement process utilizes a range of enforcement actions—from educational and remedial to punitive legal enforcement remedies, including fines and criminal sanctions in the most serious cases. When FAA determines that an immediate need exists to protect public safety, it can issue an emergency order revoking or suspending a certificate. However, although revoking a domestic repair station’s certificate is a possibility if the facility remains out of compliance, it is a time-consuming process that can often take years. In a forthcoming GAO report on FAA enforcement actions, we analyzed all 2,200 certificate actions (this figure includes airlines, repair stations, pilots, etc.) taken during fiscal year 1996 and Page 57 GAO/RCED-98-21 Aviation Safety Chapter 3 Incomplete Documentation of Inspections Makes Extent of Follow-Up Difficult to Determine found that it took an average of 13 months to close an enforcement case. Our point is not that a sizeable number of repair stations have such serious deficiencies that FAA should undertake enforcement actions. Rather, it is that certificate renewal appears to operate as a sort of “gate” that helps ensure that repair stations quickly fix their problems, big or small, thus helping to bring their operations up to the quality intended by Federal Aviation Regulations. Due process could be a concern. A headquarters official said that if FAA allows an inspector to make the decision about whether a repair station keeps or loses its certificate, it could infringe on the due process requirements afforded domestic repair station operators. However, according to FAA attorneys, due process would not be a concern. The manager of the FAA Airworthiness Law Branch explained that as currently is the case with foreign repair stations, the final decision not to renew a certificate would rest with the Administrator. The same due process rights accorded to foreign repair stations would apply to domestic repair stations. Our audit work and recent work by the Department of Transportation’s Conclusions Inspector General have identified continuing problems with the documentation of inspections and the quality of data entered into the Program Tracking and Reporting Subsystem, FAA’s tracking system. FAA’s guidance is very limited in specifying what documentation offices should keep. As a result, the documentation contained in the files on domestic repair stations, which acts as the official record of dealings between FAA and repair stations, did not contain sufficient data for us to determine how quickly and completely deficiencies found during inspections were being corrected. While FAA’s computer-based tracking system gives the agency means for overseeing the inspection program, problems with the quality of its data could jeopardize the reliability of FAA’s new computerized system, the Safety Performance Analysis System, in determining when to target greater inspection resources to repair stations that warrant more intensive oversight than others. In previous reports, we have recommended that FAA develop and implement a comprehensive strategy to improve the quality of all data used in its databases. Until FAA completes the implementation of this strategy, the extent and the impact of the problems with the quality of the system’s data will remain unclear. Data problems notwithstanding, there is some anecdotal evidence that foreign repair stations may have greater incentive than domestic repair Page 58 GAO/RCED-98-21 Aviation Safety Chapter 3 Incomplete Documentation of Inspections Makes Extent of Follow-Up Difficult to Determine stations to quickly fix deficiencies found during inspections because foreign repair stations are faced with a certificate renewal requirement and domestic repair stations are not. However, we were unable to verify this because of FAA’s poor documentation, particularly for domestic repair stations. Quick correction of all problems, large or small, helps to ensure better repair station operations, and having to do less follow-up to determine whether repair stations have taken corrective action frees FAA inspectors to conduct other work. Views within FAA vary widely as to whether a certificate renewal requirement, which appears to be at the heart of quicker compliance, should be extended to domestic repair stations. Although the evidence is not complete enough to support a recommendation that FAA take such a step, extending the certificate renewal requirement remains a potential option for consideration. To ensure that FAA inspectors are effectively documenting and resolving Recommendations deficiencies found during inspections, we recommend that the Secretary of Transportation instruct the Administrator of FAA to take the following actions: • Specify what documentation should be kept in files on repair stations to record complete inspection results and follow-up actions. The documentation should include inspection results, deficiency letters, repair station responses to deficiencies, and FAA’s responses indicating that the deficiencies were corrected. • Monitor the implementation of the strategy to improve data quality to ensure it is completed as soon as possible so that the data used in SPAS are reliable when the system is fully implemented in 1999. FAA agreed with these recommendations but did not indicate how or when Agency Comments it would implement them. Page 59 GAO/RCED-98-21 Aviation Safety Chapter 4 Actions Currently Under Way to Augment Oversight of Repair Stations Following the May 1996 crash of a ValuJet airplane in the Florida Everglades, FAA announced six initiatives to upgrade the oversight of repair stations. These initiatives are not aimed at the inspection and follow-up activities discussed in the previous chapters. Instead, the initiatives are aimed at clarifying and augmenting the oversight role of air carriers, which, under FAA regulations, share responsibility for ensuring that repair stations are qualified to do the work and are performing responsibly. FAA did not intend that these initiatives would provide for any significant improvements in FAA’s own inspections of repair stations. However, other efforts now under way, coupled with our recommendations in the previous chapters, could help address problems with current inspections. For many years, FAA officials have acknowledged that regulations governing the inspection of repair stations need to be improved. Since 1989, FAA has been in the process of revising these regulations. Though progress has been made, FAA officials remain uncertain about when the proposed regulations will be published for review and comment. Inspectors responding to our survey said overwhelmingly that revised regulations would help the oversight effort. FAA is also adding more than 700 inspectors to its ranks who will, in part, oversee repair stations. Survey responses from current inspectors indicated that the success of this effort will depend in part on the qualifications of new inspectors and on the training available to all those in the inspector ranks. Finally, FAA is revising its regulations to require that repair station mechanics receive more training. FAA announced the improvements to its inspection policies on June 18, Existing Initiatives 1996. These improvements consist of six specific initiatives designed, Are Targeted according to FAA, to “toughen the FAA’s oversight of airlines that rely on Primarily at Air contract maintenance and training.” Under Federal Aviation Regulations, air carriers share with FAA the responsibility of ensuring that repair Carriers stations are conducting work that meets safety standards. FAA is responsible for ensuring that repair stations comply with regulations; air carriers are responsible for ensuring that repair stations perform maintenance in accordance with the air carriers’ manuals. The six initiatives were issued because FAA identified problems in the way some air carriers provided oversight of repair stations, according to FAA’s Deputy Associate Administrator for Regulation and Certification. Accordingly, FAA’s initiatives focus on strengthening the oversight role of air carriers, Page 60 GAO/RCED-98-21 Aviation Safety Chapter 4 Actions Currently Under Way to Augment Oversight of Repair Stations not on modifying FAA’s own approach to inspecting repair stations and ensuring that corrective action is taken. The six initiatives involve actions to be taken by the air carriers or by FAA inspectors overseeing air carriers (see fig. 4.1). Collectively, these initiatives require that (1) air carriers demonstrate regulatory compliance for each of their contract facilities doing substantial heavy maintenance or repairs; (2) FAA ensure that air carriers list all contractors performing substantial maintenance for them; and (3) air carriers audit repair stations they want to begin using. They also call for additional review by FAA inspectors—mainly those inspectors who oversee air carriers. Page 61 GAO/RCED-98-21 Aviation Safety Chapter 4 Actions Currently Under Way to Augment Oversight of Repair Stations Figure 4.1: June 1996 Initiatives Designed to Improve Air Carrier Oversight of Repair Stations Initiative: FAA will require airlines to demonstrate the regulatory compliance of each Stricter repair of their major contract maintenance and training programs. station oversight by air carriers Initiative: FAA will review the airline's procedures to demonstrate compliance of More FAA review major contract maintenance organizations to ensure that they are included as part of of what air carriers the carrier's approved maintenance program. Also, quality assurance oversight are doing to performed by the carrier will be reviewed to ensure that the work conducted by the oversee repair contractor conforms to the maintenance program. stations Initiative: FAA will review the "check airman" involvement and on-site oversight being provided by the carrier of contract training facilities to ensure that the contractor is providing services that comply with the regulatory requirements levied upon the carrier. Initiative: FAA will create new oversight requirements for inspectors who monitor repair stations and training centers. These inspectors will be required to check not only a repair station's compliance with the regulations but also air carriers' assurance that the maintenance and repair done by the repair station are in compliance with air carriers' maintenance programs. Initiative: FAA's principal inspectors will require that carriers list all contractors Listing all repair performing substantial maintenance and training in an airline's operations stations performing specifications. substantial maintenance Initiative: Before the use of new contractors is approved by the principal inspector Air carrier audits of for addition to the operations specifications, the carrier must conduct an audit of the contractor. This audit must demonstrate to the principal inspector that the any repair stations contractor is capable of performing the contracted work in accordance with the added to the list carrier's approved programs. Page 62 GAO/RCED-98-21 Aviation Safety Chapter 4 Actions Currently Under Way to Augment Oversight of Repair Stations Because these initiatives were directed at air carriers, they had little or no effect on FAA’s direct oversight of repair stations. In fact, all 72 repair station inspectors who responded specifically to our question about the impact of FAA’s initiatives said that the initiatives and guidance have had no impact on the extent of their surveillance activities. They told us that because the initiatives were directed at the air carriers and the FAA inspectors responsible for overseeing the air carriers, rather than at the inspectors overseeing repair stations, their oversight activities have not changed. FAA Has Taken Steps to FAA implemented these initiatives through two major efforts—issuing Implement All Six guidelines and issuing a handbook bulletin. In July 1996, FAA headquarters Initiatives issued the National Flight Standards Work Program Guidelines. These guidelines stated that air carriers would be required to demonstrate that programs at each major repair station complied with regulations. The guidelines required air carriers to list all repair stations performing substantial heavy maintenance in their operations specifications.1 For the FAA inspectors whose duties involve monitoring the air carriers (as opposed to monitoring the repair stations themselves), the guidelines stressed the importance of careful oversight but did not require specific actions on the inspectors’ part. FAA headquarters officials said the guidelines left it to inspectors to decide whether to place more emphasis on the surveillance of repair stations as part of their oversight of air carriers. Matters in the guidance for inspectors to consider in making their decisions included a particular FAA office’s work demands and the complexities or problem areas of the repair stations involved. Inspectors overseeing air carriers told us these initiatives have increased their awareness of the need to oversee repair stations working for air carriers, but inspectors overseeing repair stations told us the initiatives had not changed their oversight activities. FAA augmented the guidelines a month later with a handbook bulletin that provided more specific guidance for the last two initiatives shown in figure 1 Operations specifications name the maintenance activities the repair station is authorized to perform. FAA issues a set of operations specifications during certification. Page 63 GAO/RCED-98-21 Aviation Safety Chapter 4 Actions Currently Under Way to Augment Oversight of Repair Stations 4.1. The handbook bulletin defined “substantial maintenance”2 for air carriers, thereby clarifying which repair stations needed to be included on an air carrier’s operations specifications or audited by an air carrier prior to adding them to the list. The handbook bulletin’s more specific guidance on these matters was as follows: • To implement the initiative that all repair stations performing significant maintenance be listed, FAA inspectors were to list, by October 1, 1996, the repair stations that performed substantial maintenance, ensure that each repair station had had a current audit, include the repair stations as part of the operations specifications, and update FAA’s Vital Information Subsystem database with the new operation specification information. • To implement the initiative that air carriers conduct audits of any repair stations added to the list, FAA inspectors were to ensure, effective September 1, 1996, that air carriers audit any new maintenance repair station they want to add to their operations specifications. The audits’ purpose was to ensure that the repair stations are capable of performing the contracted work in accordance with the carriers’ approved programs. Under the handbook’s procedures, FAA must also review and accept the audit before an air carrier can use the contractor. FAA officials told us about 150 air carriers needed to comply with the bulletin by identifying the repair stations performing substantial maintenance for them. The officials said all of the carriers had complied by late September or early October 1996. Subsequent to that, air carriers have added other repair stations to their operations specifications after having their audits of the repair stations approved by FAA inspectors. For example, one air carrier recently added two repair stations to its operations specifications after the FAA inspector reviewed and approved both repair station audits. In contrast, another FAA inspector responsible for an air carrier denied use of two new repair stations when he found deficiencies in the audit reports for both repair stations. FAA officials stated that the review of air carriers’ audits of repair stations is something the inspectors will be doing on an ongoing basis. Any time an air carrier wants to add a repair station that does substantial maintenance, the carrier must audit the repair station and the principal maintenance 2 FAA defined substantial maintenance as any activity involving a C-check (routine airframe maintenance) or greater maintenance; any engine maintenance requiring case separation or teardown; and/or major alterations or repairs performed on airframes, engines, or propellers. Under FAA’s definition of “substantial maintenance,” a relatively small number of repair stations are likely to be affected by these requirements. We reviewed the operations specifications for 10 air carriers that implemented FAA’s requirements. Of the more than 5,000 total vendors on these air carriers’ lists of approved vendors, 268 were classified as authorized to perform “substantial maintenance.” Page 64 GAO/RCED-98-21 Aviation Safety Chapter 4 Actions Currently Under Way to Augment Oversight of Repair Stations inspector for that air carrier will need to accept the audit before the repair station can be used. Moreover, every time a new repair station is added, the air carrier’s operations specifications must be changed. FAA headquarters officials and all of the 86 repair station inspectors we Planned Update of interviewed told us current regulations governing the oversight of repair Repair Station stations are out of date in a number of respects. According to FAA, the Regulations Is Slow in current repair station regulations are based primarily on concepts that were developed during the infancy of the aviation industry. Aircraft, power Coming plants, maintenance, alteration concepts, and technology have progressed substantially in the last three decades. However, very few substantive changes have been made to the regulations since 1962. Portions of the regulations are no longer appropriate or have become increasingly difficult to administer, while some other portions no longer make a significant contribution to aviation safety. As a result, FAA has had to grant exemptions and create special administrative procedures to handle situations not provided for adequately in the regulations. In 1975, FAA and industry officials recommended revising substantial requirements of the repair station regulations. According to FAA, minor amendments to the regulations were subsequently adopted, but no major revision was made. In 1989, in light of public meetings that were part of its regulatory review, FAA decided to revise the regulations completely. However, it has taken 8 years to prepare the revisions, FAA officials said. They attributed the delays to the project being preempted by other rulemaking and policy projects. FAA hopes to begin implementing revised regulations in the coming months. It has prepared revised regulations, and headquarters officials told us the revisions are now being reviewed by the Department of Transportation and the Office of Management and Budget. Officials did not know when the revisions would be published for comment in the Federal Register. FAA had established a target of summer 1997 for publishing these revisions, but this target was not met. FAA will consider the comments received before taking action on the proposed revisions. The Deputy Associate Administrator for Regulation and Certification anticipated that FAA would receive voluminous comments, necessitating considerable time for review and response. FAA headquarters officials outlined several things they hoped to accomplish with the proposed regulations. First, they are proposing that Page 65 GAO/RCED-98-21 Aviation Safety Chapter 4 Actions Currently Under Way to Augment Oversight of Repair Stations domestic repair stations be required to have a quality control system that is based on the Joint Aviation Authorities’ (JAA) system.3 Second, the proposed regulations simplify the repair station rating system and make the ratings less confusing than the existing system. Third, the proposed regulations impose training requirements for entry-level personnel. Existing regulations do not require that entry-level personnel be trained. They require only that repair stations use practical tests or employment records to determine the abilities of uncertified employees. Finally, the proposed regulations make repair stations responsible for controlling and evaluating their vendors. Existing regulations do not require that repair stations evaluate their subcontractors or vendors. They require only that repair stations have a method of inspecting incoming material to ensure that it is free from apparent defects or malfunctions. FAA attorneys noted that the proposed rule will remove the distinction between most domestic and foreign repair stations. There are indications that some in the repair station industry may oppose many of these changes. For example, the National Air Transportation Association, an industry association, stated that it expects that the proposed regulations will require new training programs, additional record-keeping requirements, and the implementation of quality assurance systems like the air carriers’ quality assurance systems. This, the association predicts, will increase repair station costs, causing as many as a third of them to turn in their repair station certificates. The association contends that the anticipated proposed regulations will “cripple the maintenance industry,” and it plans to fight them. This opposition indicates that completing the rulemaking process may take a significant amount of time. The importance of completing this project can be seen in inspectors’ responses to our survey. As figure 4.2 shows, most inspectors believe that various changes in the regulations would help them carry out their inspection duties. Of the inspectors we surveyed, most (88 percent) favored updating the regulations as a way to improve repair station regulatory compliance. In particular, most inspectors (77 percent) favored changes to require repair stations to notify FAA of the names of air carriers for which they do work. Most inspectors (78 percent) also favored 3 A number of European nations have banded together to coordinate their efforts through JAA. Like FAA, the national aviation authorities of JAA member nations have developed their own extensive inspection, surveillance, evaluation, and certification programs for repair stations. Under the provisions of the Bilateral Aviation Safety Agreement, FAA is negotiating agreements and procedures with several countries that will eliminate the duplicate oversight of repair facilities by FAA and such other entities. Discussions are still largely in the formative stage. Page 66 GAO/RCED-98-21 Aviation Safety Chapter 4 Actions Currently Under Way to Augment Oversight of Repair Stations changing regulations to require air carriers to provide their manuals or procedures along with the parts to be repaired by repair stations. Figure 4.2: FAA Inspectors’ Responses to Ideas for Improving Compliance 100 Percentage of respondents Through Changes Related to Existing Regulations 80 60 40 20 0 Issue update Require repair Change for repair stations to regulations to station notify FAA of require airlines to regulations the names of provide all air carriers maintenance for which they manuals or do work procedures with parts to be repaired Strongly or generally in favor Neutral Strongly or generally opposed Note: Percentages may not add to 100 because a few respondents said they had no basis to make a judgment. Inspectors provided examples of why revised regulations are needed. One inspector said the regulations do not address new repair techniques such as nondestructive testing and repair of composite materials, which means that the inspector must evaluate a repair station’s practices using his or her own judgment. Two inspectors noted that regulations require repair Page 67 GAO/RCED-98-21 Aviation Safety Chapter 4 Actions Currently Under Way to Augment Oversight of Repair Stations stations to have an FAA-approved inspection procedure manual but do not require repair stations to follow it. As a result, an inspector who finds that a repair station failed to follow its approved manual cannot write a violation. The Congress has provided FAA with additional resources to hire more Adding to the inspectors. FAA increased the number of inspectors from 2,324 in fiscal Inspector Corps and year 1994 to an estimated 3,062 at the end of fiscal year 1997, a 32-percent Improving Training increase. FAA’s approved budget for fiscal year 1995 authorized 201 additional inspectors; for fiscal year 1996, 237 additional inspectors, and for fiscal year 1997, 300 additional inspectors. To fill the additional authorizations and to rehire for attrition, FAA hired 302 inspectors in fiscal year 1995 and 361 in fiscal year 1996. About 63 percent of the inspectors hired were airworthiness inspectors. According to the manager of the Human Resource Programs Branch, most airworthiness inspectors have the oversight of repair stations as part of their duties. FAA has requested 235 additional inspectors in its fiscal year 1998 budget estimate. If the request is approved, this would represent a 42-percent increase in inspector staffing since fiscal year 1994. Responses to our survey indicate that current FAA inspectors believe the impact of these new inspectors will depend mainly on how strong the applicants’ qualifications are. As figure 4.3 shows, current inspectors believe that aviation industry experience, particularly supervisory experience, is important. For example, 81 percent of inspectors responding to our survey strongly or generally favored having inspectors with hands-on technical experience in industry as a way to improve repair stations’ compliance. FAA’s current qualifications for entry-level inspectors require maintenance experience in a repair station, air carrier facility, or military repair facility, and 3 years of supervisory experience. The manager of the Human Resources Program Branch told us FAA requires that inspectors have 3 years of supervisory experience because they must be able to communicate orally and in writing with mechanics, engineers, and managers. We did not determine whether FAA’s newly hired inspectors met the agency’s qualification standards for new hires. Page 68 GAO/RCED-98-21 Aviation Safety Chapter 4 Actions Currently Under Way to Augment Oversight of Repair Stations Figure 4.3: FAA Inspectors’ Responses to Ideas for Improving Compliance 100 Percentage of respondents Through Inspector Qualifications 80 60 40 20 0 Ensure that inspectors Ensure that inspectors have hands-on technical have supervisory experience in industry experience in industry Strongly or generally in favor Neutral Strongly or generally opposed Note: Percentages may not add to 100 because a few respondents said they had no basis to make a judgment. The training of new and existing inspectors is another area that has been a focus of attention. We and others have reported for several years that FAA’s aviation safety inspectors are not receiving needed training. Most recently, in October 1996 we issued a report recommending that FAA evaluate the impact of recent budget reductions on critical safety-related functions, including training, and report the results to the Congress through the appropriations process.4 FAA inspectors’ responses to our survey indicate that most inspectors continue to be concerned about the need for improved training. Specifically, 82 percent of the inspectors surveyed said they strongly or generally favored providing inspectors with maintenance 4 See Aviation Safety: New Airlines Illustrate Long-Standing Problems in FAA’s Inspection Program (GAO/RCED-97-2, Oct. 17, 1996). Page 69 GAO/RCED-98-21 Aviation Safety Chapter 4 Actions Currently Under Way to Augment Oversight of Repair Stations and avionics training, including hands-on training, as a way to improve repair stations’ compliance with regulations. Over three-quarters of the inspectors (80 percent) favored more training on inspection skills. Additionally, 45 percent said that the inability to get needed training is at least a minor reason why inspectors are not able to ensure repair stations’ compliance with all aspects of the regulations. These results add support for our 1996 recommendation. Inspectors also expressed their concern about inadequate training in written comments on our survey forms. For example, one inspector stated that inspectors need specific training on aircraft and systems. Another inspector wrote, “I have completed FAA repair station certification and surveillance course; however, that course does not educate in the procedures for overhaul. To understand the product, I am relying on personal experience I had before I joined FAA, with no recurrent training on the actual product.” Another inspector stated he needed more hands-on training on a turbine engine before he was sent to inspect it. Because of significant technological advances in the aviation industry, Upgrading current FAA regulations that prescribe the certification requirements for an Certification and estimated 145,000 mechanics and repairmen need to be updated. Aviation Training of Aviation maintenance is one of the most complex areas of the industry, and aviation maintenance personnel must possess many technical skills. Maintenance Changes in aircraft technology have also significantly increased the need Personnel for specialized training. FAA has been updating, consolidating, and clarifying all its certification, training, experience, and currency requirements for aviation maintenance personnel for a new rule (14 C.F.R. part 66) entitled “Certification: Aviation Maintenance Personnel.” According to FAA, some of the key features of this proposed rule include the • creation of additional certificates and ratings for aviation maintenance personnel, • expansion of current certification requirements, and • establishment of additional training and recurrent training requirements for certified aviation maintenance personnel. Initially, on August 17, 1994, the proposed rule was published in the Federal Register for public comment. However, the final issuance of the rule has been delayed because FAA officials decided that to avoid confusion they need to combine the proposed rule with other rule Page 70 GAO/RCED-98-21 Aviation Safety Chapter 4 Actions Currently Under Way to Augment Oversight of Repair Stations revisions. According to FAA headquarters officials, FAA plans to reissue the revised proposed rule in the Federal Register for public comment in December 1997. Although the various activities FAA has under way may help strengthen the Conclusions oversight of repair stations, none of them directly addresses the concerns about inspection and follow-up that we discussed in chapters 2 and 3—namely the limited success in identifying problems through reviews by individual inspectors of large facilities and the inadequate documentation of efforts to correct deficiencies found during inspections. FAA’s initiatives may help the air carriers—and the FAA inspectors who monitor those air carriers—be more attentive to the work being performed by repair stations, but they do not appear to have any direct link to improving the quality of FAA’s inspections of repair stations or the speed and thoroughness with which problems are resolved. Also, as FAA has struggled to deal with a growing workload caused by new airlines and the greater complexities of a deregulated environment, FAA has received a 32-percent increase in the number of its inspectors since fiscal year 1994. To use these additional resources as effectively as possible, FAA needs to overcome its inspection program’s weaknesses in identifying problems at repair stations and in documenting inspection results that need follow-up. The results of our work also underscore the need for progress in several areas that FAA is addressing by updating repair station regulations, hiring new inspectors, and improving training programs. Progress on initiatives for updating regulations on the oversight of repair stations and the certification and training requirements for maintenance personnel has been slow. These efforts may require additional management attention. To ensure that outdated regulations governing the oversight of repair Recommendation stations and certification and training requirements for maintenance personnel are updated as soon as possible, we recommend that the Secretary of Transportation instruct the Administrator of FAA to expedite the efforts to update the regulations and to establish and meet schedules for completing the updates. FAA agreed with the recommendation but did not indicate how or when it Agency Comments would be implemented. Page 71 GAO/RCED-98-21 Aviation Safety Appendix I Objectives, Scope, and Methodology In December 1996, the Ranking Minority Member of the Aviation Subcommittee of the Senate Committee on Commerce, Science, and Transportation, and Senator Ron Wyden asked us to examine FAA’s oversight of repair stations. Specifically, we were asked to address the following questions: • What is the nature and scope of the oversight of repair stations, conducted by FAA personnel? • How well does FAA follow up on inspections to ensure that deficiencies in repair stations’ operations are corrected once they have been identified? • What steps has FAA taken to improve the oversight of repair stations? Our analysis was based in part on agencywide data and in part on a detailed review involving a cross-section of airlines, repair stations, FAA offices, and FAA inspectors. Our industrywide data included the amount of maintenance costs incurred by airlines. For a more in-depth look at how air carriers were using repair stations, we selected eight air carriers for review, choosing them in part because, like the industry as a whole, they varied greatly in the extent to which they used repair stations (see table I.1). At the eight airlines, we discussed the type and amount of maintenance contracted to repair stations. Also, we discussed the impact of recent FAA initiatives on the air carrier and the methods each uses to oversee repair stations. Page 72 GAO/RCED-98-21 Aviation Safety Appendix I Objectives, Scope, and Methodology Table I.1: Airlines Included in GAO’s Detailed Review Percentage of Nature of maintenance maintenance Nature of performed by activity done by Airline operations repair stations repair stations Alaska Airlines Uses 74 aircraft on 10 (airframe) Airframes, power the Pacific Coast 90 (power plants) plants, radios, from Alaska to 75 (instruments instruments and Mexico and the and accessories) accessories Russian Far East America West Airlines Uses 102 aircraft 85 Airframes, power on routes plants, radios, throughout the instruments and U.S. and to Mexico accessories and Canada American Airlines Uses 659 aircraft 26 Power plants, on routes radios, instruments throughout the and accessories, world but primarily interior modifications Delta Airlines Uses 542 aircraft 10 Airframes, power on routes plants, radios, throughout the instruments and world accessories, but primarily modifications and predeparture checks Sierra Pacific Airlines Uses 3 aircraft for 90-95 Airframes, power charter and plants, radios, supplemental instruments, services on a propellers, and contractual basis accessories Simmons Airlines Uses 93 aircraft 12 Airframes, power operating as plants, and American Eagle on accessories routes in the Southwest and Midwestern U.S. Southwest Airlines Uses 242 aircraft 85 Airframes, power on routes plants, radios, throughout the U.S. instruments and accessories United Airlines Uses 591 aircraft 7 Airframes, power on routes plants, radios, throughout the instruments and world accessories, but primarily modification and routine maintenance Page 73 GAO/RCED-98-21 Aviation Safety Appendix I Objectives, Scope, and Methodology We also visited 10 repair stations, choosing them in part because they represented a variety of locations (both domestic and foreign), types of repair station activities, and size of operations (see table I.2). At these repair stations, we observed maintenance capabilities, shipping and receiving procedures, and work package documentation. We also met with quality assurance officials to talk about internal controls over parts, suspected unapproved parts, and oversight by FAA and air carrier representatives. Table I.2: Repair Stations Included in Detailed Review Repair station and location Size and extent of operations Advanced Material Technologies, Inc. Employs about 170 people and mainly Tempe, Arizona performs work on power plants and accessories AeroControls, Inc. Employs about 450 people and performs Auburn, Washington all types of maintenance work except power plants B.F. Goodrich Component Services Division Employs about 20 people and works Tempe, Arizona exclusively on wheels and brakes; one of a number of B.F. Goodrich repair stations Chromalloy Los Angeles Employs about 205 people and works Gardena, California exclusively on power plants; one of a number of Chromalloy repair stations Evergreen Air Center Employs about 590 people and performs Marana, Arizona all types of work except instruments Greenwich Air Services, Inc. Employs about 840 people and mainly Miami, Florida performs work on power plants and accessories Lufthansa Technik AG Employs about 3,300 people and performs Frankfurt, Germany routine and contract maintenance for Lufthansa and other carriers on airframes, power plants, and accessories Precision Avionics and Instruments, Inc. Employs about 35 people and works Atlanta, Georgia exclusively on electrical and electronic instruments, components, and accessories Sabena Technic Employs about 1,600 people and performs Brussels National Airport, Belgium work for Sabena and other carriers in all maintenance areas Tramco, Inc. Employs about 2,200 people and mainly Everett, Washington performs work on airframes and accessories; one of several B.F. Goodrich repair stations We conducted reviews at FAA headquarters in Washington, D.C., four of the nine FAA regional offices (Northwest Mountain, Southern, Southwest, and Western-Pacific), 8 of FAA’s 86 Flight Standard district offices (Atlanta, Page 74 GAO/RCED-98-21 Aviation Safety Appendix I Objectives, Scope, and Methodology Dallas, Dallas/Fort Worth, Fort Worth, Los Angeles, Miami, Scottsdale, and Seattle), and six of the seven international offices located inside and outside the United States (Brussels, Dallas/Fort Worth, Frankfurt, London, Miami, and San Francisco). We selected the regional and Flight Standards district offices because of their geographical diversity and because the locations were responsible for overseeing both domestic and foreign repair stations. We reviewed FAA’s criteria, procedures, and other documents used to oversee domestic and foreign repair stations and discussed them with 86 principal maintenance and avionics inspectors responsible for overseeing repair stations and 17 inspectors responsible for air carriers that contract with repair stations. We also conducted detailed reviews of a judgmental sample of 488 repair station files to determine if inspection and follow-up activity was documented effectively. In addition, we analyzed data from FAA’s computer-based Program Tracking Reporting Subsystem for fiscal years 1993 through 1996 to determine if FAA effectively captured repair station inspection results. To gather information about the experiences of FAA inspectors overseeing repair stations and their views on ways to improve the oversight of repair stations, we conducted a mail survey. We mailed our survey to a sample of 275 of the 601 FAA inspectors listed in FAA’s Vital Information Subsystem as being responsible for the oversight of one or more repair stations doing work for part 121 or 135 carriers during fiscal year 1996. In total, we received responses from 247 inspectors, a response rate of 90 percent. Of these, 202 completed the questionnaire, while 45 did not because they did not oversee any repair stations in fiscal year 1996. (See app. II of this report for additional information on the survey methodology and app. III for the summary results of that survey.) The 202 respondents were responsible for overseeing a total of 1,375 repair stations that performed maintenance for part 121 or 135 carriers, with the number of repair stations per inspector ranging from 1 to 37. Table I.3 shows the sizes of repair stations overseen by the inspectors responding to our survey. Table I.3: Distribution of Repair Stations Overseen by FAA Inspectors, Size of repair station (number of by Size employees) Percentage of total More than 100 17 16-100 35 15 or fewer 48 Total 100 Note: Percentages can be generalized to the nearly 2,800 repair stations doing work for air carriers. Page 75 GAO/RCED-98-21 Aviation Safety Appendix I Objectives, Scope, and Methodology We also interviewed officials from the government agencies that are responsible for reviewing FAA’s oversight of repair stations and air carriers: the Office of the Inspector General, U.S. Department of Transportation; the Air Carrier Survey and Analysis Office, Air Mobility Command Staff, U.S. Air Force, Department of Defense; and the National Transportation Safety Board. We conducted our review from August 1996 through October 1997, in accordance with generally accepted government auditing standards. Page 76 GAO/RCED-98-21 Aviation Safety Appendix II Survey Methodology To examine the experiences and opinions of the FAA employees who inspect repair stations, we surveyed inspectors within FAA’s Flight Standards Service. We mailed our survey to a random sample of inspectors listed in FAA’s Vital Information Subsystem database as conducting inspections of repair stations that do work for air carriers. The survey asked for such information as the degree of success inspectors have in identifying all deficiencies, factors that hinder inspectors’ success in ensuring that repair stations comply with all aspects of the regulations, and ways to improve the compliance of repair stations. Our response rate for the survey was 90 percent. A summary of the responses is in appendix III. FAA’s database indicated that 601 inspectors were responsible for repair stations that do work for air carriers. We drew a random sample of 275 inspectors. This sample size was designed to provide sampling errors of no more than 5 percent at the 95-percent confidence level. Since we used a sample (called a probability sample) of 275 FAA inspectors to develop our estimates, each estimate has a measurable precision, or sampling error, that may be expressed as a plus/minus figure. A sampling error indicates how closely we can reproduce from a sample the results that we would obtain if we were to take a complete count of the universe using the same measurement methods. By adding the sampling error to and subtracting it from the estimate, we can develop upper and lower bounds for each estimate. This range is called a confidence interval. Sampling errors and confidence intervals are stated at a certain confidence level—in this case, 95 percent. (See table II.1.) For example, a confidence interval, at the 95-percent confidence level, means that in 95 out of 100 instances, the sampling procedure we used would produce a confidence interval containing the universe value we are estimating. Page 77 GAO/RCED-98-21 Aviation Safety Appendix II Survey Methodology Table II.1: Sampling Errors for Selected Percentages Percentage Sampling error 5 ±2 10 ±3 20 ±4 30 ±5 40 ±5 50 ±5 60 ±5 70 ±5 80 ±4 90 ±3 95 ±2 Note: Sampling errors are calculated for the 95-percent confidence level using the finite population correction factor and 185 cases, the smallest number of valid cases for questions with finite categories. In addition, table I.3 in appendix I requires ratio estimates in order to calculate sampling errors. Table II.2 shows sampling errors for these estimates. Table II.2: Sampling Errors for Repair Stations by Number of Employees Sampling error Size of repair station (percentage added or (number of employees) Percentage subtracted)a More than 100 16.6 ±2.7 16-100 35.3 ±3.5 15 or fewer 48.1 ±4.1 Total 100.0 n/a a Sampling errors calculated at the 95-percent confidence level. We conducted 11 pretests of our survey with inspectors in two of FAA’s nine regions, including 3 pretests with inspectors of foreign repair stations and 8 pretests with inspectors of domestic repair stations. Each pretest consisted of a visit with a single FAA employee by two or three GAO staff. The pretest attempted to simulate the actual survey experience by asking the employee to fill out the questionnaire while the GAO staff observed and unobtrusively took notes. Then the employee was interviewed about the questionnaire items to ensure that (1) the questions were readable and clear, (2) the terms were precise, (3) the survey did not place an undue Page 78 GAO/RCED-98-21 Aviation Safety Appendix II Survey Methodology burden on FAA employees that would result in a lack of cooperation, and (4) the survey appeared independent and unbiased in its point of view. We modified the final survey based on our pretesting results. In addition to our pretesting, we had managers in FAA’s Flight Standards Service in Washington, D.C., review our questionnaire, and we obtained comments from a survey research psychologist at FAA’s Training and Organizational Research Laboratory in Oklahoma City and from officials of the union representing the FAA inspectors we surveyed. We incorporated comments from these reviews as appropriate. During the pretesting phase, it became evident that respondents considered the survey questions to be sensitive. Specifically, some respondents said less experienced inspectors might be afraid of retribution if their answers were made public. To address these concerns, we developed procedures to guarantee the complete anonymity of all survey responses. To do this, we did not retain any identification of the respondent on the survey booklet or return envelope. This procedure prevented us from knowing the identity of the respondent for any of the surveys returned to us. The use of a separate return postcard allowed us, nevertheless, to track which respondents did and did not mail back survey responses so that we could follow up with those who did not respond. Pretest respondents told us these measures would encourage inspectors to return the survey forms. To increase the response to our survey, we mailed a prenotification letter to respondents 1 week before we mailed the survey on January 31, 1997. We also used three mailings after the survey mailing: (1) a reminder postcard 1 week after the survey, (2) a reminder letter to nonrespondents 18 days after the survey, and (3) a replacement survey for recipients not yet responding mailed 4 weeks after the survey. We received the last survey included in our analysis on March 20, 1997. We received survey responses from 247 inspectors for a response rate of 90 percent. Of those responding, 202 inspectors were assigned to oversee repair stations that did work for air carriers during fiscal year 1996. Only these respondents actually filled out our survey and are included in our survey results in this report. Table II.3 shows a summary of the survey returns. Page 79 GAO/RCED-98-21 Aviation Safety Appendix II Survey Methodology Table II.3: Summary of Returns to Mail Surveys Number of FAA inspectors Population size 601 Total sample size 275 Surveys returneda 247 Eligible 202 Not eligible 45 Surveys not returned 28 Response rate (number returned/number 90% mailed) a Does not include surveys returned that were not filled out. Page 80 GAO/RCED-98-21 Aviation Safety Appendix III Survey Questions and Responses Page 81 GAO/RCED-98-21 Aviation Safety Appendix III Survey Questions and Responses Page 82 GAO/RCED-98-21 Aviation Safety Appendix III Survey Questions and Responses Page 83 GAO/RCED-98-21 Aviation Safety Appendix III Survey Questions and Responses Page 84 GAO/RCED-98-21 Aviation Safety Appendix III Survey Questions and Responses Page 85 GAO/RCED-98-21 Aviation Safety Appendix III Survey Questions and Responses Page 86 GAO/RCED-98-21 Aviation Safety Appendix III Survey Questions and Responses Page 87 GAO/RCED-98-21 Aviation Safety Appendix III Survey Questions and Responses Page 88 GAO/RCED-98-21 Aviation Safety Appendix III Survey Questions and Responses Page 89 GAO/RCED-98-21 Aviation Safety Appendix III Survey Questions and Responses Page 90 GAO/RCED-98-21 Aviation Safety Appendix IV Major Contributors to This Report Robert Jerry Aiken Leslie Albin John H. Anderson, Jr. Steven N. Calvo Fran A. Featherston Thomas A. Kai Christopher M. Jones Julia A. Rachiele David A. Robinson Stanley G. Stenerson Mindi G. Weisenbloom (341498) Page 91 GAO/RCED-98-21 Aviation Safety Ordering Information The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. VISA and MasterCard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent. Orders by mail: U.S. General Accounting Office P.O. Box 37050 Washington, DC 20013 or visit: Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000 or by using fax number (202) 512-6061, or TDD (202) 512-2537. Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone. A recorded menu will provide information on how to obtain these lists. For information on how to access GAO reports on the INTERNET, send an e-mail message with "info" in the body to: email@example.com or visit GAO’s World Wide Web Home Page at: http://www.gao.gov PRINTED ON RECYCLED PAPER United States Bulk Rate General Accounting Office Postage & Fees Paid Washington, D.C. 20548-0001 GAO Permit No. G100 Official Business Penalty for Private Use $300 Address Correction Requested
Aviation Safety: FAA Oversight of Repair Stations Needs Improvement
Published by the Government Accountability Office on 1997-10-24.
Below is a raw (and likely hideous) rendition of the original report. (PDF)