United States Government Accountability Office GAO Report to Congressional Committees September 2012 AVIATION SAFETY Additional FAA Efforts Could Enhance Safety Risk Management GAO-12-898 September 2012 AVIATION SAFETY Additional FAA Efforts Could Enhance Safety Risk Management Highlights of GAO-12-898, a report to congressional committees. Why GAO Did This Study What GAO Found The nation’s aviation system is one of The Federal Aviation Administration (FAA) and its business lines and offices are the safest in the world, but with air in different stages of their implementation of Safety Management Systems travel projected to increase over the (SMS). FAA finalized its agency-wide implementation plan in April 2012, and the next 20 years, efforts to ensure the Air Traffic Organization (ATO) has completed its SMS implementation, but other continued safety of aviation are FAA SMS efforts are in the early stages. FAA business lines, such as the increasingly important. The FAA is Aviation Safety Organization (AVS) and the Office of Airports (ARP), have SMS seeking to further enhance safety by guidance and plans largely in place and have begun to integrate related practices shifting to a data-driven, risk-based into their operations, but many implementation tasks remain incomplete, and safety oversight approach—referred to officials and experts project that full SMS implementation could take many years. as SMS. SMS implementation is required for FAA and several of its There are a number of key practices that can help agencies plan for and business lines and the agency is taking efficiently implement new projects, including large scale transformations such as steps to require industry FAA’s SMS implementation, and FAA has many in place. For example, FAA has implementation. support from top leadership and a clear project mission. However, FAA has only As requested, this report addresses partially addressed other key practices such as developing a project plan to track (1) the status of FAA’s implementation SMS implementation, and FAA has not addressed performance-related practices of SMS, (2) the extent to which FAA’s such as establishing SMS performance measures or links between employees’ SMS efforts have been consistent with performance standards and SMS. key practices for successful planning and implementation of a new program, Alignment of FAA’s SMS Implementation with Key Practices and (3) challenges FAA faces in implementing SMS. To address these issues, GAO reviewed FAA SMS documents, compared FAA efforts to key practices, and interviewed agency and industry officials. What GAO Recommends GAO recommends that FAA develop systems to: track SMS implementation, evaluate employee performance as it relates to SMS, and assess whether SMS meets its goals and objectives; conduct a workforce analysis for SMS; and consider strategies to address airports’ data concerns. The Department of Transportation agreed Several challenges remain that may affect FAA’s ability to effectively implement to consider the recommendations and SMS. FAA is taking steps to address some challenges and stakeholder provided clarifying information about concerns, but challenges related to data sharing and data quality; capacity to SMS, which GAO incorporated. conduct SMS-based analyses and oversight; and standardization of policies and procedures could negatively affect FAA’s efforts to implement SMS in a timely and efficient manner. Further, FAA officials stated that SMS implementation will require some skills that agency employees do not have, but FAA has not yet assessed the skills of its workforce to identify specific gaps in employee expertise. In addition, while existing federal law protects any data collected for View GAO-12-898. For more information, SMS, any data airports collect could be subject to state-specific Freedom of contact Gerald L. Dillingham, Ph.D., at (202) 512-2834 or firstname.lastname@example.org. Information Act laws, a gap that could create a disincentive for airports to fully participate in SMS implementation. United States Government Accountability Office Contents Letter 1 Background 4 FAA and Its Business Lines Are at Different Stages of SMS Implementation 8 FAA’s SMS Approach Is Consistent with Many but Not All Key Practices for Successful Project Implementation 15 Addressing Key Implementation Practices and Other Challenges Could Enhance SMS Effectiveness 24 Conclusions 32 Recommendations for Executive Action 33 Agency Comments 33 Appendix I Objectives, Scope, and Methodology 35 Appendix II GAO Contact and Staff Acknowledgments 38 Table Table 1: Airport Interviews 37 Figures Figure 1: The Four Components of Safety Management Systems 5 Figure 2: Organizational Structure to Coordinate SMS Implementation across FAA Business Lines and Offices 7 Figure 3: FAA’s SMS Guidance Links to International and Government-wide SMS Requirements and Guidance 9 Figure 4: Timeline of Key SMS Implementation Activities across FAA’s Business Lines and Offices, 2007 to 2013 11 Figure 5: Proposed and Potential Future Applicability of Federal Aviation Regulation Part 5 to the Aviation Industry 13 Figure 6: Alignment of FAA’s SMS Implementation with GAO- Identified Key Practices for Successful Organizational Transformations 16 Page i GAO-12-898 Safety Management Systems Abbreviations AIR Aircraft Certification Service AFS Flight Standards Service ANG Office of NextGen ARP Office of Airports AST Office of Commercial Space Transportation ATO Air Traffic Organization AVP Office of Accident Investigation and Prevention AVS Aviation Safety Organization CFR Code of Federal Regulations FAA Federal Aviation Administration FOIA Freedom of Information Act ICAO International Civil Aviation Organization JPDO Joint Planning and Development Office NTSB National Transportation Safety Board SMS Safety Management System SRM Safety Risk Management This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. Page ii GAO-12-898 Safety Management Systems United States Government Accountability Office Washington, DC 20548 September 12, 2012 The Honorable John D. Rockefeller, IV Chairman The Honorable Kay Bailey Hutchison Ranking Member Committee on Commerce, Science, and Transportation United States Senate The Honorable John L. Mica Chairman The Honorable Nick J. Rahall, II Ranking Member Committee on Transportation and Infrastructure House of Representatives The Honorable Thomas E. Petri Chairman The Honorable Jerry F. Costello Ranking Member Subcommittee on Aviation Committee on Transportation and Infrastructure House of Representatives The U.S. airspace system is one of the safest in the world, with no main line air carrier passenger fatalities in the U.S. in more than 10 years. 1 This record reflects the efforts of the Federal Aviation Administration (FAA), airlines, airports, manufacturers, the National Transportation Safety Board (NTSB), and others to continually improve aviation safety. However, with air travel projected to increase over the next 20 years, efforts to ensure the continued safety of aviation are increasingly important. FAA is attempting to further enhance aviation safety, in part, by shifting to a data- driven, risk-based safety oversight approach. This approach is becoming the standard throughout the global aviation industry and is recognized by aviation leaders such as the International Civil Aviation Organization 1 Main line air carriers are commercial airlines that use jets with over 90 seats, as compared to regional carriers, which use smaller piston, turboprop, and regional jet aircraft with up to and including 90 seats. Page 1 GAO-12-898 Safety Management Systems (ICAO), the Joint Planning and Development Office (JPDO), and others as the next step in the evolution of safety. 2 FAA is overseeing implementation of this new approach—called a safety management system (SMS) approach—both within FAA and throughout the U.S. aviation industry and is coordinating these efforts with the international aviation community. Safety management systems represent a proactive approach to safety and are intended to continually monitor all aspects of aviation operations and collect appropriate data to identify emerging safety problems before they result in death, injury, or significant property damage. Under SMS, FAA will use the aviation safety data it collects to identify conditions that could lead to aviation accidents or incidents and to address such conditions through changes in the FAA’s organization, processes, management, and culture. SMS adoption and implementation is one of the biggest cultural and procedural transformations in FAA history and will likely involve years of continuous effort on the part of agency and industry officials. You asked us to assess FAA’s implementation of SMS. To do so, we addressed the following questions: (1) What is the status of FAA’s implementation of SMS? (2) To what extent have FAA’s SMS efforts been consistent with key practices for successful planning and implementation of a new program? (3) What challenges does FAA face in implementing SMS? To determine the status of FAA’s implementation of SMS, we reviewed FAA’s SMS orders and pilot project guidance, implementation plans, and Notices of Proposed Rulemaking for Part 121 air carriers and Part 139 2 ICAO is an agency of the United Nations that promotes the safe and orderly development of international civil aviation worldwide. ICAO sets standards and regulations necessary for aviation safety, security, and efficiency and serves as a forum for collaboration among its 191 member states. JPDO was established by the U.S. Congress in 2003 to plan and coordinate research and development for the Next Generation Air Transportation System (NextGen). Pub. L. No. 108-176, §709. JPDO manages partnerships that include agencies such as the Departments of Transportation, Defense, and Homeland Security. Page 2 GAO-12-898 Safety Management Systems airports. 3 We interviewed FAA SMS program managers across FAA business lines and offices. We also reviewed international and FAA guidance and SMS and NTSB recommendations to FAA related to SMS. To assess the extent to which FAA’s efforts have been consistent with key practices, we reviewed our reports and other literature on successful project planning and implementation, particularly for large-scale transformative projects, 4 and identified key practices applicable to FAA’s SMS implementation. We then assessed FAA’s actions against the key practices by examining such documents as FAA’s guidance and implementation plans, and interviewing FAA officials. We determined whether each key practice was addressed, partially addressed, or not addressed by using criteria developed from prior GAO reports. For example, we considered a practice partially addressed if FAA had partially implemented, taken steps toward, or started but not completed implementing it (see app. I for a discussion of our assessment). To identify challenges FAA faces in implementing SMS, we reviewed our prior work on long-standing FAA challenges, such as those related to training and data, and interviewed aviation industry experts and FAA officials. We also reviewed our prior work on performance measurement and workforce analysis, and NTSB recommendations related to SMS. To obtain industry views on challenges, we interviewed representatives from airports and air carriers selected for size and geographic dispersion. We also interviewed representatives of aviation industry associations and reviewed written comments submitted by aviation stakeholders on two FAA-issued Notices of Proposed Rulemaking that would require SMS for some airports and air carriers. See appendix I for a more detailed description of our scope and methodology. We conducted this performance audit from September 2011 to September 2012 in accordance with generally accepted government 3 FAA issues operating certificates to air carriers offering scheduled, commercial air carrier service under 14 CFR Part 121. For the purposes of this report, we will refer to these carriers as commercial air carriers. FAA issues airport operating certificates to airports that (1) serve unscheduled air carrier aircraft with more than 30 seats; or (2) serve scheduled air carrier operations in aircraft with more than 9 seats under 14 CFR Part 139. 1(a)(1) and (2). For the purposes of this report, we will refer to these as certificated airports. 4 See GAO, Results-Oriented Cultures: Implementation Steps to Assist Mergers and Organizational Transformations, GAO-03-669 (Washington, D.C.: July 2, 2003) and GAO, Motor Carrier Safety: The Federal Motor Carrier Safety Administration Has Developed a Reasonable Framework for Managing and Testing Its Comprehensive Safety Analysis 2010 Initiative, GAO-08-242R (Washington, D.C.: Dec. 20, 2007). Page 3 GAO-12-898 Safety Management Systems auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. SMS provides a top-down approach to managing safety risk, which FAA Background expects will improve aviation safety. SMS is not an additional safety program that is distinct from existing activities that accomplish an entity’s safety mission, but rather, a process for safety management that incorporates systematic procedures, practices, and policies. According to FAA, the overarching goal of SMS is to improve safety by helping ensure that the outcomes of any management or system activity incorporate informed, risk-based decision making. We reported in 2010 that FAA officials believe that successfully implementing SMS is critical to meeting the challenges of a rapidly changing and expanding aviation system. To achieve a higher level of safety in an already very safe system, FAA requires a more forward-thinking approach, which SMS provides, by addressing cultural and organizational problems that lead to safety hazards, identifying system-wide trends in aviation safety, and managing emerging hazards before they result in incidents or accidents. SMS implementation should bring about a fundamental shift in aviation safety oversight. For decades, the aviation industry and federal regulators, including FAA, have used data reactively to identify the causes of aviation accidents and incidents and take actions to prevent their recurrence. While FAA plans to continue to use data to analyze past safety events, it is also working to use data proactively to search for risks. FAA’s shift to the proactive approach of SMS is important because, as accidents have become increasingly rare, less information is available for reactive analyses of their causes. As a result, information that can be used to help identify accident and incident precursors has become more critical for accident prevention. Thus, the open sharing of safety information among aviation stakeholders and how FAA’s policies and procedures govern the reporting of safety information are essential to the success of SMS. SMS consists of four key components: (1) safety policy, (2) safety risk management, (3) safety assurance, and (4) safety promotion (see fig.1). Together, these four components are intended to provide a systematic approach to achieving acceptable levels of risk. FAA provides to its personnel detailed guidance on the principles underpinning these Page 4 GAO-12-898 Safety Management Systems components and the application of these components to aviation oversight in its official orders and other internal FAA guidance. To the industry, FAA provides this SMS guidance via advisory circulars and a dedicated page for the SMS program office on the FAA website. Figure 1: The Four Components of Safety Management Systems FAA is undertaking the transition to SMS in coordination with the international aviation community, working with ICAO to adopt applicable global standards for safety management. ICAO requires SMS for the management of safety risk in air operations, maintenance organizations, air traffic services, and airports as well as certain flight-training operations and for organizations that design or manufacture aircraft. Further, ICAO has published safety management requirements for its member countries that mandate that civil aviation authorities—such as FAA—establish SMS. 5 ICAO first mandated SMS worldwide for air traffic service providers, such as air carriers and certified aerodromes, in 2001. ICAO later specified that member states should mandate SMS implementation for airports, air carriers, and others by 2009. FAA began SMS implementation in 2005, but FAA officials informed ICAO that the agency and industry would not be able to meet the 2009 deadline. ICAO is 5 ICAO refers to SMS as employed by civil aviation authorities as “State Safety Programs.” Page 5 GAO-12-898 Safety Management Systems allowing FAA to take additional time in its efforts to implement SMS, with the understanding that implementation is under way and that FAA is in the midst of a rulemaking to require SMS for commercial air carriers. ICAO officials stated that the United States is one of the leading implementers of SMS worldwide and acknowledged that SMS implementation in the U.S. aviation system may be more complicated than in other countries because of the size and complexity of the U.S. aviation industry. ICAO has not specified a date by which FAA is expected to comply with the requirements to implement SMS in the aviation system. There have also been actions within the United States to encourage implementation of SMS. For instance, in 2007, NTSB recommended that FAA require all commercial air carriers to establish an SMS 6 and, in 2011, added SMS for all modes of transportation to the NTSB’s Most Wanted List, identifying SMS as one of the most critical changes needed to reduce the number of accidents and save lives. Partially in response to the ICAO requirement, FAA added goals related to SMS implementation to its 2009-2013 Flight Plan. 7 These are linked to a requirement to implement SMS in three of FAA’s business lines—the Air Traffic Organization (ATO), the Aviation Safety Organization (AVS), and the Office of Airports (ARP)—and a goal to implement SMS policy in all appropriate FAA organizations, which include the Office of Commercial Space Transportation (AST) and the Office of NextGen (ANG). FAA is in the process of implementing SMS within these business lines and offices as well as in industry through rulemakings to require airports and commercial air carriers to implement SMS. FAA designated AVS as the lead for SMS implementation in September 2008. Within AVS, the Office of Accident Investigation and Prevention’s (AVP) Safety Management and Research Planning Division coordinates and manages SMS implementation and operation across the agency, and so AVP serves as the official SMS lead for the agency. FAA has also established groups that work across the agency to coordinate the agency’s implementation of SMS. The FAA SMS Committee is comprised of managers from each of the four business lines and one staff office currently implementing SMS, and reports to the FAA SMS Executive Council, which is composed of Associate and Assistant 6 NTSB recommendation A-07-010. 7 Federal Aviation Administration, 2009-2013 Flight Plan is the agency’s strategic plan. Page 6 GAO-12-898 Safety Management Systems Administrators, their deputies, and other high-level FAA officials from each business line or office (see fig. 2). Within some of the business lines, there are offices devoted to specific aviation oversight functions that are responsible for overseeing detailed implementation of SMS for those functions. For example, the Flight Standards Service (AFS), a division of AVS that provides safety oversight of commercial air carriers and others, is taking steps to require SMS implementation by commercial air carriers and is also working to integrate SMS into its internal activities. In addition, the Aircraft Certification Service (AIR), a division of AVS that provides safety oversight to aviation design and manufacturing firms, is leading agency efforts to encourage SMS implementation for that industry sector, while ARP is leading agency efforts to require SMS implementation for certificated airports. Figure 2: Organizational Structure to Coordinate SMS Implementation across FAA Business Lines and Offices SMS implementation will require changes to many of FAA’s operations. As the agency and industry implement SMS, shifts will be necessary in both the skills of FAA and industry staff and the tools that the agency uses to monitor safety. FAA’s integration of SMS into its business practices will also affect how the agency provides air navigation services and oversees the aviation industry. Historically, FAA oversight of airlines, airports, and other regulated entities has involved oversight of such things as operations and maintenance. FAA will continue this oversight, but will also apply SMS principles to its processes for oversight. The agency will provide oversight of the safety management systems of service providers Page 7 GAO-12-898 Safety Management Systems such as air carriers and airports to help ensure that they are managing safety within their operations through SMS. For example, AFS currently provides oversight of the operations, maintenance, and safety data of commercial air carriers and others. Once SMS is fully implemented, AFS will continue to provide this oversight and will also conduct oversight of the safety management systems that commercial air carriers and others put in place. ATO completed its implementation of SMS, but FAA and several of its FAA and Its Business other business lines and offices are in the early stages of implementation. Lines Are at Different Most FAA business lines and offices have guidance and plans for SMS implementation in place and have begun to integrate SMS-related Stages of SMS practices into their operations, but many tasks remain and aviation Implementation officials and experts with whom we spoke project that full SMS implementation will take many years. FAA Recently Finalized Its FAA finalized its agency-wide plan for SMS implementation in April 2012. Agency-Wide The plan provides a road map for SMS implementation across the agency Implementation Plan, but and describes the activities that FAA business lines and offices will need to complete by the end of 2015 to integrate SMS into their operations. 8 Full SMS Implementation These activities will lead to outcomes including: Is Likely to Take Many Years • revising and standardizing safety policies and safety risk management methodologies across FAA to ensure SMS principles are consistently addressed; • improving organizational processes so that FAA business lines and offices can share safety data and information more easily; and • coordinating communications to ensure a common understanding of SMS across the agency. FAA began its agency-wide SMS implementation efforts in 2008, and in September of that year issued a policy for implementation of a common SMS within FAA. Among other things, the policy sets forth management 8 The year 2015 does not represent an estimated completion date for SMS implementation. The implementation plan describes tasks through 2015, but FAA officials estimate SMS implementation efforts will continue beyond 2015. Page 8 GAO-12-898 Safety Management Systems principles to guide all of FAA in safety management and safety oversight activities and requires AVS, ARP, and ATO to develop and execute business line-specific plans for SMS implementation. In late 2008, FAA formed the agency-wide FAA SMS Committee to coordinate implementation efforts across FAA business lines and offices. Overall, the agency has taken a bottom-up approach to implementation, with some individual business lines and offices beginning implementation prior to agency-wide efforts. FAA has also taken steps to ensure that its plans for SMS implementation and policies align with international and government-wide requirements and technical guidance on SMS implementation, including ICAO’s Standards and Recommended Practices, the ICAO Safety Management Manual, and the JPDO SMS Standards. For instance, officials stated that they consulted international and government-wide guidance on SMS implementation when drafting agency implementation plans. (See fig. 3 for more information on alignment of FAA requirements with international and government-wide requirements and guidance on SMS.) Figure 3: FAA’s SMS Guidance Links to International and Government-wide SMS Requirements and Guidance Although FAA has made progress, completion of SMS implementation across FAA is likely to take many years. FAA’s agency-wide SMS implementation plan includes tasks with estimated completion dates Page 9 GAO-12-898 Safety Management Systems through 2015, and some implementation tasks may take even longer to complete. For instance, a project plan that AVS officials developed to track status of AVS SMS implementation tasks contained in its implementation plan includes task completion dates through 2016. According to FAA, the overall SMS implementation effort is an evolutionary process that will not have a specific completion date. The current implementation time frame is consistent with experts’ estimates of how long it may take to implement SMS and with other large-scale organizational transformations. For example, representatives from The MITRE Corporation, which manages a federally funded research center for FAA and assisted FAA in selected SMS implementation efforts, stated that organizational transformations like SMS can take from 6 to 10 years. ATO Completed SMS ATO is the only entity among FAA and its business lines to have Implementation in 2010 completed SMS implementation. ATO issued its internal SMS guidance in March 2007 and finalized both its SMS implementation plan and its updated SMS Manual in 2008. 9 According to ATO officials, ATO completed SMS implementation in March 2010, and the FAA Air Traffic Safety Oversight Service validated that ATO’s implementation of SMS was complete. Officials stated that implementation within ATO was simpler, in part, because it is the only branch of FAA that is considered an aviation service provider and therefore did not have to conduct a rulemaking for external entities as part of its SMS implementation. 10 With the implementation phase complete, ATO is currently in the continuous improvement phase of SMS. This means that ATO will continuously use the SMS-based processes now in place to identify hazards, enact strategies to mitigate the risks associated with those hazards, and assess the extent to which the mitigations are working effectively. In addition, FAA officials stated that ATO is working to improve its SMS operations, will update guidance on SMS, and plans to perform audits of its SMS functions on a regular basis. ATO officials added that they are working to 9 As part of FAA’s bottom-up approach to SMS implementation, ATO began implementation in 2005, much earlier than other FAA business lines. Officials stated that the agency did not initially intend to implement SMS across the entire agency. 10 The primary service of ATO is to move air traffic safely and efficiently. The controllers, technicians, engineers, and support personnel employed by ATO provide air navigation services directly to commercial and private aviation stakeholders, as well as the military. Since ATO directly employs those providing the aviation service, it was not necessary for FAA to conduct a rulemaking in order to require ATO to implement SMS. Page 10 GAO-12-898 Safety Management Systems share lessons learned from their implementation efforts with other FAA business lines and to develop SMS tools and processes that can be commonly implemented across all FAA business lines. Most FAA Business Lines With the exception of ATO, most FAA business lines and offices are in and Offices Are in Early the early stages of implementation, either in terms of integrating SMS into Stages of Implementation their internal processes or in terms of their efforts to prepare to provide oversight for proposed requirements for industry implementation of SMS. To date, much of the work of the FAA business lines has focused on efforts to draft implementation policies and guidance, train employees, and create tools for applying safety analyses and risk-based decision- making to safety oversight. (See fig. 4 for more information on the status of key SMS implementation efforts across FAA.) Figure 4: Timeline of Key SMS Implementation Activities across FAA’s Business Lines and Offices, 2007 to 2013 Page 11 GAO-12-898 Safety Management Systems Aviation Safety Organization AVS began its SMS implementation efforts in August 2006 and finalized its SMS implementation plan in January 2012, which was then incorporated into FAA’s overall plan for SMS implementation. Since 2006, AVS and its seven services and offices have issued orders and other guidance on SMS implementation; developed SMS training courses; conducted voluntary pilot projects and rulemaking efforts on SMS implementation for industry; and worked to begin integrating elements of SMS into their operations. For example, AIR officials, who provide oversight of aviation design and manufacturing firms, have developed a central database that provides standard criteria for analyzing service data in a risk-based manner. This should allow AIR inspectors and engineers to rate the risk of potential safety issues and prioritize oversight to high risk issues. Some services and offices within AVS are in the midst of efforts to require SMS for industry and are also operating voluntary pilot programs to promote SMS implementation within industry. A final rule to require SMS for commercial air carriers is expected to be issued in September 2012. 11 In 2007, AFS launched a pilot program to encourage voluntary implementation of SMS by industry. 12 According to FAA officials, as part of its rulemaking efforts for commercial air carriers, FAA and AVS are developing a new part in the Code of Federal Regulations (CFR)—Part 5—that will describe SMS implementation requirements for Part 121 certificate holders. In the future, FAA may conduct rulemakings to require additional sectors of the aviation industry to meet Part 5 requirements (see fig. 5). AVS officials stated that 11 The Airline Safety and Federal Aviation Administration Extension Act of 2010 requires FAA to issue a final rule for Part 121 air carriers by July 30, 2012. Pub. L. No. 111-216, §215(c)(2), 124 Stat. 2348, 2366 (2010). According to FAA officials, the rule will be applied to Part 121 operators as well as operators with dual certificates that conduct operations under both 14 CFR Parts 121 and 135. The proposed rule will not apply to operations conducted solely under 14 CFR Part 135. FAA officials stated that the proposed rule is currently being reviewed by the Office of the Secretary of Transportation and the Office of Management and Budget prior to final issuance. 12 The AFS pilot project included Part 121 air carriers, Part 135 operators, and Part 145 repair stations. FAA regulations set forth certification procedures for aviation-related products and parts under Federal Aviation Regulation Part 21, commuter and on-demand aviation operations under Federal Aviation Regulation Part 135, and repair stations under Federal Aviation Regulation Part 145. Page 12 GAO-12-898 Safety Management Systems efforts to establish SMS requirements more broadly across the aviation industry will likely take many years. Figure 5: Proposed and Potential Future Applicability of Federal Aviation Regulation Part 5 to the Aviation Industry Though FAA has not yet required SMS for air carriers or other parts of industry, FAA has acted to encourage SMS implementation by industry through voluntary pilot projects, and some aviation stakeholders have chosen to implement SMS in advance of any federal requirement. Some sectors of the aviation industry are farther along in their implementation of SMS than others. For instance, FAA officials stated that a large majority of commercial air carriers are in the process of implementing SMS. 13 As of June 2012, over 90 percent of commercial air carriers operating under Part 121 were participating in the AFS pilot program, which provides air carriers with direct implementation support from FAA officials under a more relaxed implementation time frame than is anticipated under an eventual implementation regulation. Of these air carriers, three have reached the final stage of SMS implementation. 14 However, most small air carriers have not yet begun implementing SMS. In contrast to AFS, AIR is at an earlier stage in its efforts to require SMS for the approximately 13 Part 135 operators also hold air carrier certificates, but according to FAA officials, a smaller proportion of these operators are participating in the AFS pilot project. 14 FAA does not certify SMS implementation by commercial air carriers participating in the pilot project as complete. However, FAA has laid out four phases of SMS implementation for pilot project participants, and FAA officials meet with participants to verify whether or not they have completed the steps included in each phase. Page 13 GAO-12-898 Safety Management Systems 3,000 design and manufacturing firms it oversees. AIR began a voluntary pilot project for SMS implementation by design and manufacturing firms in 2011 and has 11 pilot project participants. AIR officials stated that they are in the process of launching a second aviation rulemaking committee to continue to explore options to require SMS for design and manufacturing firms. Officials also noted that AFS and AIR are working together to share lessons learned and assist one another in their implementation efforts. Office of Airports ARP is in the early stages of working to integrate SMS principles into its oversight of airports, and recently took steps to reduce the scope of that oversight. ARP initially planned to apply SMS-based oversight to all certificated airports. Officials stated that ARP is currently limiting its SMS- based oversight to large hub airports because of budget constraints and will reassess its capacity to expand oversight to smaller airports in 2013. ARP began its SMS implementation in 2010 and issued an internal order to provide a basis for the integration of SMS into its operations later that year. The office finalized its SMS implementation plan in September 2011 and has begun to make changes to its oversight. For instance, In June 2011, ARP began to apply SMS-based oversight to construction projects at the 29 large hub airports in the United States. 15 Under this new oversight framework, ARP staff assess proposed airport construction projects using risk-based SMS principles, and airports need to incorporate strategies to mitigate identified risks into their construction plans prior to receiving ARP’s approval for the project. Like AVS, ARP is also in the midst of a rulemaking to require SMS for all certificated airports and has completed three voluntary SMS pilot projects for airports from 2008 to 2011. 16 Thirty-one airports participated in at least one of ARP’s SMS pilot projects. ARP is using information gathered through the pilot projects to inform a planned advisory circular that will provide additional guidance to airports on SMS implementation. The pilot projects also allowed airports to share their SMS implementation practices with other airports. The final rule to require SMS for Part 139 certificated 15 Large hub airports are those which enplane at least 1 percent of U.S. passenger enplanements system-wide. See 49 U.S.C. § 47102 (10). 16 The first pilot project focused on airport creation of SMS implementation plans; the second pilot project focused on SMS implementation for smaller airports, and the third pilot project focused on SMS implementation at airports that participated in at least one of the first two pilot projects. Page 14 GAO-12-898 Safety Management Systems airports is expected to be issued in April 2013 and, if implemented as proposed, would require over 500 airports to implement SMS. 17 Offices of Commercial Space Other FAA business lines are in varying stages of implementation. AST is Transportation and NextGen not currently required to implement SMS; however, AST is taking initial steps toward integrating SMS into an existing set of safety management processes. ANG is farther along in its implementation of SMS because of its previous status as a part of ATO. 18 According to officials, ANG is basing its implementation of SMS on policies and processes established during ATO’s implementation of SMS. The officials stated that since ANG will provide the systems and components that will be used by ATO to manage air traffic, it made sense for ANG to develop its SMS based on policies, processes, and systems established by ATO. Officials stated that ANG completed its implementation plan in June 2012 and estimated that ANG’s SMS implementation is about 70 percent complete. There are a number of key practices and implementation steps that can FAA’s SMS Approach help agencies successfully plan for and implement new projects, including Is Consistent with large scale transformative ones, such as FAA’s implementation of SMS. As we have previously reported, addressing these key practices can help Many but Not All Key an agency improve its efficiency, effectiveness, and accountability. 19 FAA Practices for currently has many of these key factors in place, such as established Successful Project support from top leadership and a clear project mission; however, it has only partially addressed other key practices, such as providing needed Implementation expertise and technology, and has yet to establish SMS performance measures (see fig. 6). 17 While ARP has limited its SMS-based oversight efforts to large hub airports, the proposed rulemaking would require Part 139 certificated airports of all sizes to integrate SMS into their operations. 18 In 2011, FAA reorganized some of its offices and, as part of the reorganization, separated NextGen efforts from ATO. 19 GAO-03-669 and GAO-08-242R. Page 15 GAO-12-898 Safety Management Systems Figure 6: Alignment of FAA’s SMS Implementation with GAO-Identified Key Practices for Successful Organizational Transformations Many Key Practices Are in FAA has instituted many key practices that will help it prepare for and Place implement SMS across its business lines and offices. • Top leadership: Top leaders from each FAA business line provide support for and actively participate in SMS implementation. As previously mentioned, FAA established the SMS Executive Council, a group of high-ranking FAA officials that provides executive-level guidance and conflict resolution for SMS-related issues across the Page 16 GAO-12-898 Safety Management Systems agency. In accordance with our key practices, the SMS Executive Council has the authority to make resource allocation decisions, but also confers decision-making authority where appropriate to the FAA SMS Committee. For instance, FAA officials told us that the SMS Executive Council retains the authority to make final decisions about changes to FAA’s implementation plan that affect policies or procedures for multiple business lines; the FAA SMS Committee has the authority to make decisions that relate to daily concerns that fall within the purview of its members. For example, committee members settled a disagreement between ATO and airport officials over whether an airport should conduct certain components of a safety risk management panel. At the time, FAA had not yet issued its safety risk management policy clarifying terms and requirements, so the airport and ATO each had its own distinct safety risk management definitions and processes. Working with ARP and ATO officials, committee members identified a compromise in which ATO protocols were followed, but any disagreements on terms or procedures were documented. ARP officials told us that FAA’s safety risk management policy, issued in April 2012, should help prevent this type of disagreement from occurring. • Clear project mission: FAA’s internal order requiring SMS implementation for ARP, ATO, and AVS clearly describes that FAA’s mission is to improve aviation safety and that implementing SMS and its components supports that mission. Each business line also has its own internal order requiring SMS implementation that mirrors this mission and goals. • Implementation team: AVP’s safety management division and the FAA SMS Committee, function jointly as FAA’s dedicated SMS implementation team. The team’s structure and actions align with our criteria for a strong and stable team 20 because it is composed of senior-level program managers from each business line, all of whom 20 GA0-03-669. Page 17 GAO-12-898 Safety Management Systems had received SMS training according to FAA officials. Despite some recent departures, its membership has been largely stable. 21 • Leading practices: FAA shares information across business lines to identify lessons learned related to SMS implementation. For example, ATO assembled lessons learned from its SMS implementation into a presentation for the other business lines, and included tips such as encouraging others to implement a training program and monitor mitigations. According to FAA’s implementation plan, the agency plans to systematize the sharing of lessons learned by creating a central repository to collect and communicate safety lessons learned among its business lines and offices by September 30, 2013. • Troubleshooting: FAA has processes in place to manage SMS implementation across FAA, including troubleshooting unexpected problems. For example, the FAA SMS Committee meets monthly and manages agency-wide SMS implementation and any challenges that arise, and regularly briefs the SMS Executive Council, a briefing that includes a discussion of any issues or unexpected problems that could not be resolved at the committee level. For instance, when the Air Traffic Manager at an airport disagreed with airport officials regarding how to handle a potential safety issue with planes that were taking off on runways that were temporarily closed, the FAA SMS Committee elevated the issue to the SMS Executive Council, which resolved it. As we have previously reported, instituting practices like these can help an agency become more results-oriented, customer- focused, and collaborative. Although FAA is still in the process of finalizing new requirements for airports and air carriers to implement SMS, it has already taken some steps to institute key practices for those efforts. For example, FAA officials stated that the agency has taken steps to identify leading practices during pilot projects by soliciting information from participating 21 Our review of implementation practices focused on FAA’s internal implementation efforts; however, FAA does not currently have a dedicated implementation team to coordinate efforts to require industry SMS across FAA business lines. FAA officials stated that the FAA SMS Committee is the appropriate group to handle industry SMS implementation issues that cross FAA organizations, but committee officials told us that SMS implementation in industry is implemented separately by AFS and ARP. In addition, FAA reported that several FAA offices, including the Rulemaking Management Council and the Office of Rulemaking, manage all of FAA’s rulemaking efforts. Page 18 GAO-12-898 Safety Management Systems airports and air carriers, and FAA officials told us they plan to incorporate these lessons learned into rulemaking and guidance. ARP officials reported that they encouraged pilot project participants to share lessons learned directly with one another through studies and roundtable discussions, and incorporated some of the lessons learned into FAA advisory circulars. FAA has also made efforts to troubleshoot and manage unexpected problems with pilot participants through meetings, calls, and conferences with airport and air carrier officials to understand their experiences. For example, AFS officials reported that they helped officials from air carriers to understand when certain safety risk management documentation and processes are necessary, and how they could be adapted for a variety of changes made to carrier operations, including smaller day-to-day changes. However, despite this assistance, officials from some airports that participated in pilot projects reported that they could have benefited from additional assistance from ARP, such as clarification on the safety risk management component of SMS. In addition, an official at one airport told us that he would have liked FAA to facilitate conversations between airports of similar size to help them share lessons learned. FAA Efforts to Create a Other steps FAA has taken in its SMS implementation efforts partially Project Plan and Consult align with key practices for implementing a new program. and Assist Stakeholders • Project plan: Currently, the agency-wide project plan for SMS Partially Align with Key implementation is a single page of high-level milestones, which AVP Practices officials monitor and report on to the SMS Executive Council. Also, AVS has a detailed project plan for its own SMS implementation and elements of agency-wide implementation for which AVP, as the agency SMS lead, has responsibility. Officials stated that they have plans to develop a system to monitor and track the progress of activities needed to implement SMS, but FAA does not currently have a system for tracking agency-wide SMS implementation, a key practice particularly important during the initial planning phase of project implementation. 22 However, given the scope and complexity of SMS, a detailed, agency-wide project plan could help FAA track and monitor the interim steps of SMS implementation across the agency. 22 Our review of implementation practices focused on FAA’s internal SMS implementation efforts. However, FAA also does not currently have an agency-wide project plan to track its efforts to oversee industry implementation. Page 19 GAO-12-898 Safety Management Systems Without such a plan, it may be more difficult for FAA to identify problems or deviations from planned activities, putting both the timeliness and effectiveness of SMS implementation at risk. • Consulting with stakeholders: FAA has made efforts to consult with employees and stakeholders regarding its SMS implementation, but it has not yet developed a communications plan. Agencies should involve employees in planning, and incorporate employee feedback into new policies and procedures. 23 FAA involved its business line program managers and some of the managers’ staff by assigning them responsibility for the day-to-day tasks related to implementing SMS across the agency. FAA has involved other employees by soliciting questions and comments on SMS in town hall meetings and the online DOT site called “IdeaHub,” and by offering SMS training through each business line. ATO, ARP, and AVS all offer introductory SMS courses for their staff as well as additional related courses, such as an SMS course specifically for managers and ATO’s safety risk management course. FAA has been working to implement SMS for the last 4 years, but the agency does not have a communications plan or strategy for ensuring that the SMS messages communicated to staff are consistent across the agency. Instead, FAA relies on a more informal communications structure in which each program manager staffed to the implementation team communicates relevant information back to their respective business line. The implementation team does not communicate any information directly to employees, which could hinder the team’s ability to ensure consistency in its message across FAA. ATO officials reported experiencing this challenge at the beginning of ATO’s SMS implementation, when a lack of clear requirements for communicating SMS information resulted in variation in staff’s understanding of guidance. We have previously reported that a communication plan or strategy can ensure consistency of message, provide information to meet the specific needs of employees, encourage two-way communication, and build trust. FAA plans to begin working on a communications plan in September 2012, and is scheduled to issue the plan at the end of February 2013. FAA officials also said they are in the process of developing an internal SMS website for employees to share information and ideas, which could enhance SMS communications. However, until the communications plan is developed 23 GAO-03-669. Page 20 GAO-12-898 Safety Management Systems and implemented, FAA’s employees may not receive timely or consistent information on SMS or be as invested in its implementation as they might otherwise be. FAA’s approach to overseeing industry SMS implementation allowed for additional two-way communication. 24 For example, FAA solicited views on SMS implementation from airport and air carrier officials through voluntary pilot projects described previously, and learned more about industry perspectives through the formal rulemaking process—whereby an agency issues a Notice of Proposed Rulemaking and is required to notify the public and give them an opportunity to submit comments. • Providing technology and expertise: FAA has provided some SMS training and tools to its employees; however, it has not yet provided other tools important for SMS implementation. FAA officials reported that each business line has provided SMS training to staff. In addition, FAA recently developed a standardized Safety Risk Management (SRM) policy, which will assist employees across FAA by standardizing SRM terminology and clarifying confusion on the conduct of SRM across the agency. FAA plans to create a simple version of an agency-wide hazard-tracking system in the next 3 to 6 months, but does not have plans to create a more complex system until August 2015, according to FAA’s SMS implementation plan. The simple version will draw from hazard-tracking systems already in place in some business lines, and summarize information from them to highlight broader hazards such as those that would affect multiple business lines. For instance, FAA officials stated that if ATO wanted to make a change to its operations at a particular airport, then ATO would be responsible for identifying associated hazards, risks, and risk mitigations and would also be responsible for assuming responsibility for the risk. However, if ATO determined that the airport was better equipped to mitigate the identified risks, then the airport and ARP would become more involved in designing risk mitigations and overseeing their implementation. FAA’s efforts to provide tools to help in SMS implementation are affected by differences in how data are collected and assessed across the agency. For example, these differences have held back agency efforts to model how changes to the national airspace system, such as increases to air travel, can 24 Our review of implementation practices focused on FAA’s internal SMS implementation efforts, so we do not comment on FAA’s overall communication strategy. Page 21 GAO-12-898 Safety Management Systems affect safety. We have previously reported on and made recommendations related to FAA’s data challenges, 25 and also discuss them later in this report. These data challenges mean that FAA is not always able to perform comparisons across databases, a challenge that that limits the usefulness of the data in identifying possibly dangerous hazards. Identifying, monitoring, and mitigating hazards is a key tenet of SMS, and without the proper technologies and tools, FAA may not be able to do this as effectively. FAA Has Yet to Integrate FAA’s efforts do not align with two key practices for implementing a new SMS into Employee program. Performance Plans and • Integrating SMS into employee performance plans: FAA does not Establish Performance consistently evaluate employees’ performance on SMS-related tasks. Measures We have previously reported that effective performance management systems create a clear linkage between individual performance and organizational success, and include aligning individual performance expectations with organizational goals. 26 FAA’s organizational mission and goal, and that of SMS, is to improve safety, yet FAA officials told us that the agency does not require employee performance plans to include SMS-related tasks. Although officials reported that some employees’ performance plans explicitly include SMS items, such as providing SMS training or developing SMS policy, it is left to the discretion of each business line whether SMS items are included. FAA officials told us that SMS principles and methodologies will be included in the performance plans of employees involved in writing SMS policy and revising SMS processes, and will be incorporated into the tasks of others once SMS implementation reaches those individuals. However, currently, none of the business lines require this. As such, FAA does not have a system for assessing the extent to which staff are effectively supporting SMS, and FAA may not be able to determine if staff are completing tasks and responsibilities necessary for the successful implementation of SMS. 25 GAO, Aviation Safety: Improved Data Quality and Analysis Capabilities Are Needed as FAA Plans a Risk-Based Approach to Safety Oversight, GAO-10-414 (Washington D.C.: May 6, 2010). 26 GAO, Results-Oriented Culture: Creating a Clear Linkage Between Individual Performance and Organizational Success, GAO-03-488 (Washington D.C.: Mar. 14, 2003). Page 22 GAO-12-898 Safety Management Systems • Measuring performance: FAA does not have performance measures in place to assess whether the SMS goals of improving safety are being achieved. FAA has broader safety-related performance measures, such as tracking rates of runway incursions and losses of separation, 27 but SMS-related performance measures could address intermediate safety issues, such as precursors to incursions or incidents. Such measures could help FAA track progress toward its broader safety measures. 28 FAA officials told us that AVS is a member of the Safety Management International Collaboration Group, a group formed in 2009 to address safety management-related topics, including performance measures. Most recently, FAA formed an agency-wide working group to study performance metrics for SMS implementation, and FAA’s implementation plan states that such metrics will be finalized in October 2014. However, FAA officials we spoke with acknowledged that they are at the very beginning phase of this process and, although already in the process of implementing SMS, have not yet identified metrics to measure safety results under an SMS system. 29 We have previously reported that performance information is critical for achieving results and maximizing the return on federal funds. 30 Performance measures should help FAA identify the extent to which SMS implementation will contribute to increased aviation safety—FAA’s stated overall goal for SMS—as well as help identify what changes could be made to improve SMS performance over time. 27 Runway incursions are the unauthorized presence of an airplane, vehicle, or person on the runway. Losses of separation involve a loss of the minimum required distance between aircraft or as individual aircraft fly too close to terrain or obstructions. 28 We previously reported that such measures could help the Transportation Security Administration track its progress in securing transit and passenger rail systems. For more information, see GAO, Surface Transportation Security: TSA Has Taken Actions to Manage Risk, Improve Coordination, and Measure Performance, but Additional Actions Would Enhance its Efforts, GAO-10-650T (Washington, D.C.: Apr. 21, 2010). 29 We will discuss the challenges of developing SMS performance measures later in this report. 30 The Government Performance and Results Act of 1993 instituted a government-wide requirement that agencies set goals and report annually on performance, and encouraged federal agencies to conduct in-depth program evaluations to assess their program’s impact or learn how to improve results. Pub. L. No. 103-62. Congress subsequently passed the GPRA Modernization Act of 2010 (GPRAMA) which establishes a new framework aimed at taking a more crosscutting and integrated approach to focusing on results and improving government performance. Pub. L. No. 111-352, 124 Stat. 3866 (2011). Page 23 GAO-12-898 Safety Management Systems As previously mentioned, FAA has taken steps to address many of the Addressing Key practices associated with planning and implementing a new program. Implementation However, we identified six challenges that could negatively affect FAA’s efforts to implement SMS in a timely and efficient manner: Practices and Other Challenges Could 1) the large scope and complexity of SMS implementation, Enhance SMS Effectiveness 2) resource and capacity constraints, 3) standardization of policies and processes, 4) data sharing and protection, 5) data quality and usefulness, and 6) development of performance measures to evaluate SMS effectiveness. Large Scope and Implementing SMS is one of several major initiatives FAA has under way, Complexity of SMS and its sheer scope and complexity could affect, or be affected by, concurrent FAA efforts such as NextGen or Unmanned Aircraft Systems. 31 SMS requires changes in many of FAA’s operations: from the way the agency tracks hazards to the way it oversees industry. SMS will also require a transformation of FAA’s and the aviation industry’s safety culture to one in which information and safety data are shared openly, and errors are addressed through whatever action is necessary to prevent them from happening in the future. FAA is making efforts to move toward this new approach to safety, for instance by using data-sharing systems that are protected from public disclosure to encourage voluntary reporting of safety issues and enable more robust analysis of safety data among FAA and air carriers. Moreover, as previously stated, each of FAA’s business lines has its own role in implementing SMS that must be coordinated across the agency. This is particularly challenging because the business lines are at different stages of implementation and, according to FAA officials, have historically operated independently. 31 Unmanned Aircraft Systems are remotely operated aircraft and vehicles that come in a variety of shapes and sizes, and serve diverse purposes. GAO is currently conducting a study on Unmanned Aircraft Systems. Page 24 GAO-12-898 Safety Management Systems The scope and complexity of SMS implementation may also be a challenge for the aviation industry, and some stakeholders expressed concerns both in interviews and in official comments on FAA’s Notices of Proposed Rulemaking that eventual FAA requirements to implement SMS need to allow for variation in airport and air carrier operations. For example, officials from some smaller airlines and airports noted that SMS implementation could require additional resources, such as staff and software, which may not be readily available. In addition, officials from some airports and air carriers were concerned that FAA’s final requirements would be too prescriptive to allow entities to implement an SMS program that best fit their organizational type, management practices, and resources. Most stakeholders and experts we interviewed stated that FAA could design SMS requirements for airports and airlines that are scalable and flexible to accommodate this variation, which would address these concerns. For instance, airport officials from smaller airports told us that staff size limits their ability to assign a dedicated SMS employee or safety director, while some officials at larger airports said they were able to hire a SMS safety director or already had an established safety director in place. Also, FAA’s SMS implementation pilot project for airports found that 35 percent of participants planned to hire additional staff to support SMS and 15 percent were not sure. FAA officials have noted that they understand these scalability concerns, and are taking them into consideration as they develop final SMS rules for industry. Resources and Capacity SMS implementation across FAA will require some skills that agency employees currently do not have, yet FAA has not formally assessed the skills of its workforce to identify any gaps in the expertise required to implement SMS or determined how to fill those gaps. In addition, FAA officials stated that existing staff may not be able to be trained to fill SMS implementation needs in all cases. For instance, FAA officials noted that SMS implementation will require some engineers and other technical employees to understand certain terminologies and have certain knowledge, skills, and abilities, such as an enhanced ability to perform complex modeling and analysis of aviation safety data to identify potential safety hazards. AVS officials stated that to implement SMS, additional employees with skills in analyzing data for hazards and associated risks would be needed, along with additional training for existing staff. ARP officials stated that the office might need program analysts with specific Page 25 GAO-12-898 Safety Management Systems data analysis skills to implement SMS. 32 ARP officials stated that they do not expect to receive significantly more resources and, as previously mentioned, have already had to reduce the scope of the office’s SMS- based oversight because of insufficient staff. Stakeholders and experts also questioned whether FAA currently has the resources and capacity needed to fully implement SMS. For example, experts noted that FAA may not have the requisite engineers and other staff to participate in safety risk management efforts, or FAA inspectors to oversee individual airport and air carrier SMS programs. Despite these concerns, FAA has not yet conducted a strategic workforce assessment to accurately determine the skills and staffing levels it needs to manage SMS. Although FAA’s SMS implementation plan recommends that business lines create such staffing analyses, none have done so. Nor has FAA conducted an agency-wide workforce assessment for SMS. Our internal control standards state that agencies should ensure that skill needs are continually assessed to ensure workforces have the skills necessary to help the agency meet its goals. 33 We have reported that strategic workforce planning is an integral part of human capital management and helps an agency, among other things, determine the critical skills and competencies that will be needed to achieve current and future programmatic results, and then develop strategies tailored to address any gaps identified. 34 A workforce analysis could help FAA determine how to best address its most critical needs in ways that account for budget limitations, such as through retraining or shifting staff, rather than hiring additional employees. Without conducting an agency-wide SMS workforce analysis, FAA cannot be sure that it has sufficient staff, skills, or competencies to implement SMS, thus putting its SMS implementation efforts at risk. 32 FAA officials stated that ARP received approval to hire 26 additional staff in fiscal year 2012. However, most hiring is on hold until enactment of the fiscal year 2013 budget, when ARP officials will be able to assess the office’s ability to continue to fund the positions. 33 GAO, Internal Control: Standards for Internal Control in the Federal Government, GAO/AIMD-00-21.3.1 (Washington D.C.: Nov. 1999). 34 GAO, Human Capital: Key Principles for Effective Strategic Workforce Planning, GAO-04-39 (Washington D.C.: Dec. 11, 2003) and GAO/AIMD-00-21.3.1. Page 26 GAO-12-898 Safety Management Systems Standardization of SMS SMS standardization across FAA business lines and offices is central to Policies and Processes implementation success, yet developing common systems for distinct FAA business lines and offices has proved challenging. For example, FAA realizes that the agency needs a common hazard-tracking system in order to maximize SMS effectiveness, yet FAA officials and stakeholders stated that it is difficult to develop such a system because each of FAA’s business lines uses different hazard-related terms and definitions, and often different data systems. These differences, in turn, prevent the agency from performing simple comparisons across databases and have delayed advances in using data analysis to proactively identify potential safety hazards. FAA officials stated that the agency has recently taken steps to make its databases interoperable, and also recently issued a standardized policy for the safety risk management component of SMS. Both of these steps may enhance FAA’s hazard-tracking and analysis capabilities. The agency is also working with ICAO to address issues related to standardization, such as adopting a collaborative approach to increase the sharing of safety information internationally. Industry officials are also concerned that FAA inspectors and certificate management offices 35 may have different interpretations of SMS and other regulations. We and others have previously reported that variation in FAA’s interpretation of standards for certification and approval decisions is a long-standing issue. 36 Industry stakeholders we interviewed expressed concerns that a similar result could occur once final rules are issued requiring airports and air carriers to implement SMS, and could lead to airports or air carriers of similar size being held to different standards of SMS implementation. FAA officials acknowledged that this is a challenge for the agency and noted that the agency plans to provide additional training to inspectors related to oversight of SMS. Additionally, based on our 2010 recommendation, recent legislation directs FAA to 35 FAA Certificate Management Offices are located throughout the U.S. and employ staff who specialize in the certification, surveillance, and inspection of major air carriers and Flight Safety Training Centers. 36 GAO, Aviation Safety: Certification and Approval Processes Are Generally Viewed as Working Well, but Better Evaluative Information Needed to Improve Efficiency, GAO-11-14 (Washington, D.C.: Oct. 7, 2010); Congressional Research Service, U.S. Airline Industry Issues and Role of Congress, 7-5700, RL34467 (July 29, 2008). Page 27 GAO-12-898 Safety Management Systems establish an advisory body of government and industry representatives to address the issue of inconsistent interpretation of regulations. 37 FAA’s organizational structure for SMS implementation may pose challenges to standardization as well. For example, as previously mentioned, AVP’s safety management division is the lead for SMS, and AVP and the FAA SMS Committee share responsibility for implementing SMS across the agency. Despite AVP’s role as lead for SMS implementation, it does not have any additional authority compared to the other business lines’ committee representatives, something that AVP officials noted can make SMS implementation difficult. This could slow decision-making, particularly around issues that require business lines to come to a single decision, such as how to standardize policies. Nevertheless, FAA officials acknowledged that having to collaborate to implement an agency-wide SMS has improved communication among the business lines. FAA will likely continue to face challenges standardizing its policies and processes as standardization of this scale is not something the agency has previously undertaken, and the need to negotiate solutions across FAA business lines could take time. Data Sharing and Airport officials’ concerns about sharing and protecting their safety data Protection may reduce SMS effectiveness by limiting the ability of airports and FAA to analyze safety data and identify trends. Although FAA has some data protections in place, such as those established by the FAA Modernization and Reform Act of 2012, which protects data that airports and air carriers submit to FAA for SMS from federal Freedom of Information Act (FOIA) requests, 38 any data airports collect and any data air carriers share with airports could be subject to state-specific FOIA laws. Most certificated U.S. airports are either owned by a state, a subdivision of a state, or a local government body, and thus are subject to state laws, including state FOIA laws. This means that data airports collect and submit to FAA for SMS—such as information on hazards or other safety data—is protected from federal FOIA public disclosure requests, but, according to officials and experts, may be subject to public disclosure under state FOIA laws. Air carriers are not directly subject to state FOIA laws because they are privately owned. Nevertheless, officials and experts stated that these laws 37 Pub. L. No. 112-95, §313, 126 Stat. 11, 67 (2012). 38 Pub. L. No. 112-95, §310, 126 Stat. 11, 64 (2012). Page 28 GAO-12-898 Safety Management Systems could affect air carriers because any data they choose to share with airports could then be subject to state FOIA laws. As a result, air carrier officials told us they may be less likely to share safety information with airports. Airport and airline officials’ primary concern is that the public disclosure of such information could result in negative publicity or expose them to legal liability in the event of an incident or accident. FAA officials said that data protection and legal liability are two of the major concerns throughout the aviation industry that could hinder the implementation of SMS. FAA officials told us that they intend to continue to promote and expand safety information sharing efforts, but that airports could find ways to structure their SMS implementation so that they realized safety benefits while limiting the public release of air carrier safety information. In FAA’s official response to comments on two Notices of Proposed Rulemaking, FAA stated that airport officials are best situated to understand how to comply with state laws. Nonetheless, we found consensus among NTSB and many aviation stakeholders that FAA should seek congressional action regarding the protection of airport data from state FOIA laws. Data sharing can also be challenging within FAA. In 2011, we recommended that FAA improve information sharing among its programs because not doing so could limit the ability of FAA and others to analyze safety data and understand safety trends. 39 The Department of Transportation agreed that it must continue to promote and expand safety information sharing efforts and safety practices in order to maximize the effectiveness of safety data mining 40 to analyze trends and prioritize safety efforts to address hazards before they lead to incidents or accidents. However, our recommendation remains open. According to officials, ICAO has also formed the Safety Information Exchange Study Group to help enhance data protection and identify potential international solutions. 39 GAO, Aviation Safety: Enhanced Oversight and Improved Availability of Risk-Based Data Could Further Improve Safety, GAO-12-24 (Washington, D.C: Oct. 5, 2011). 40 Safety data mining involves the use of data based applications to look for hidden patterns in groups of data that can be used to predict future behavior or occurrences that may lead to an incident or accident. Page 29 GAO-12-898 Safety Management Systems Data Quality and Long-standing issues with data quality and usefulness could negatively Usefulness affect FAA’s understanding of aspects of the safety of the aviation industry and, consequently, affect SMS’s effectiveness. Obtaining relevant data and understanding how to analyze those data to identify potential hazards are major challenges that FAA will need to overcome. In recent GAO reports, we commented on FAA’s lack of data to effectively assess aviation trends for certain types of events and the safety performance of certain industry sectors. 41 For instance, in April 2012, we reported that for such events as runway excursions (when an aircraft veers off or overruns a runway) and ramp accidents (incidents or injuries that occur off the runway), a shortage of FAA data exists for analysis. 42 The Department of Transportation concurred with this and our recommendations, and stated that the agency has taken steps to improve to its data quality and usefulness. For example, the FAA SMS Committee directed a working group to determine what safety data the agency is going to collect and track and to recommend what kind of system will be needed. However, FAA has not yet fully implemented several of our recommendations aimed at improving its capability to use data for aviation safety oversight, or several data-related NTSB recommendations from recent years. For example, we recommended that FAA extend standard quality controls, as appropriate, to the databases that support aviation safety oversight to ensure that the data are as reliable and valid as possible. By not fully addressing these challenges and recommendations, FAA’s ability to comprehensively and accurately assess and manage hazards and risk will be compromised, reducing the ability of SMS to prevent incidents and accidents. SMS Performance The aviation community has widely acknowledged that developing SMS Measures performance measures is difficult, but without them, FAA will not be able to gauge the direct impact of SMS on aviation safety. Some stakeholders told us about ways in which SMS improved their organization’s operations, and these examples could provide insight into possible SMS performance measures. For instance, some airports and air carriers that participated in FAA’s SMS pilot projects reported that SMS 41 GAO-12-24 and GAO, Aviation Safety: FAA Is Taking Steps to Improve Data, but Challenges for Managing Safety Risks Remain, GAO-12-660T (Washington, D.C.: Apr. 25, 2012). 42 GAO-12-660T. Page 30 GAO-12-898 Safety Management Systems implementation improved communication across their organizations, helped them identify organizational gaps—such as those in internal auditing and training—and decreased employees’ injuries, aircraft damages, and insurance costs. Officials from the Flight Safety Foundation 43 recommended that the extent to which SMS informs management decision making, such as by redirecting resources or shifting priorities, may be one way to measure SMS effectiveness. An FAA official suggested that performance measures could be directed to specific components of SMS, for instance tracking the number of risks mitigated as a measure of safety risk management efficacy. We have previously reported that agencies need to set quantifiable outcome-based performance measures for significant agency activities, such as SMS, to demonstrate how they intend to achieve their program goals and measure the extent to which they have done so. 44 Performance measures allow an agency to track its progress in achieving intended results, which can be particularly important in the implementation stage of a new program such as SMS. In our prior work we recommended that agencies develop methods to accurately evaluate and measure the progress of implementation, and develop contingency plans if the agency does not meet its milestones to complete tasks. 45 FAA has established a working group to study the issue and participates on two international performance measures work groups: the Safety Management International Collaboration Group and the aforementioned Safety Information Exchange Study Group. 43 The Flight Safety Foundation is an independent, nonprofit, international organization engaged in research, auditing, education, advocacy, and publishing to improve aviation safety. 44 See GAO, The Results Act: An Evaluator’s Guide to Assessing Agency Annual Performance Plans, GAO/GGD-10.1.20; (Washington, D.C.: April 1998); VA Health Care: VA Should Better Monitor Implementation and Impact of Capital Asset Alignment Decisions, GAO-07-408 (Washington, D.C.: Mar. 21, 2007); NextGen Air Transportation System: FAA’s Metrics Can Be Used to Report on Status of Individual Programs, But Not of Overall NextGen Implementation or Outcomes, GAO-10-629 (Washington, D.C. July 27, 2010); Motor Carrier Safety: More Assessment and Transparency Could Enhance Benefits of New Oversight Program, GAO 11-858 (Washington, D.C.; Sept. 29, 2011). 45 GAO, FAA Airspace Redesign: An Analysis of the New York/New Jersey/Philadelphia Project, GAO-08-786 (Washington, D.C.: July 31, 2008). Page 31 GAO-12-898 Safety Management Systems FAA is making progress implementing SMS, both within the agency and Conclusions for the aviation industry. However, SMS implementation represents a significant cultural and procedural shift in how the agency will conduct business internally and provide oversight to aviation stakeholders such as air carriers and airports, and by all estimates, this transformation will take many years to complete. Going forward, if FAA is to attain the full benefits of SMS, it will be important for the agency to remain committed to fully implementing SMS across its business lines. FAA has taken a number of steps that align with practices we identified as important to successful project planning and implementation, but has not addressed or has only partially addressed other key practices. These practices are important for large-scale transformative projects such as SMS, which require a dramatic shift in FAA’s approach to safety oversight and management. In the absence of these key practices, it may be difficult for FAA to prioritize projects or monitor SMS implementation and progress toward improving safety. Aviation safety is a shared responsibility among FAA, air carriers, airports, and others in the aviation industry, and efforts to improve safety will require the agency to overcome several challenges. The magnitude of SMS’s potential impact on aviation oversight and the complexity of implementation are both a benefit and a drawback for FAA, as SMS implementation could help ensure the continued safety of the U.S. aviation system, but could also affect implementation time frames for other large initiatives as the agency works in a resource-limited environment. FAA officials believe that SMS implementation will require some skills that employees do not currently have; however, FAA has not conducted an agency-wide workforce assessment. With agency resources and capacity in great demand, it will be important for the agency to maximize the efficiency of SMS implementation, both through efficient use of its workforce and creation of policies and systems that standardize and streamline implementation. In addition, data protection concerns from airport officials and others could prevent aviation stakeholders from fully embracing SMS implementation, thus hindering its effectiveness. Without assurance of protection from state FOIA laws, some aviation stakeholder may choose to collect only the bare minimum of safety-related data or may choose to limit the extent to which collected information is shared among aviation stakeholders. The agency also lacks sufficient data to effectively assess aviation trends for some events as well as the safety performance of certain industry sectors. The ability of FAA to identify safety risks, develop mitigation strategies, and measure outcomes is hindered by limited access to complete and meaningful data. Page 32 GAO-12-898 Safety Management Systems To enhance the effectiveness of efforts to implement SMS and maximize Recommendations for the positive impact of SMS implementation on aviation safety, we Executive Action recommend that the Secretary of Transportation direct the FAA Administrator to take the following five actions: 1. To better evaluate the effectiveness of the agency’s efforts to implement SMS, develop a system to assess whether SMS meets its goals and objectives by identifying and collecting related data on performance measures. 2. To align strategic goals with employee efforts, develop a system to evaluate employees’ performance as it relates to SMS. 3. To better manage implementation, develop a system to track and report on SMS implementation across business lines. 4. To better leverage existing resources and facilitate SMS implementation, conduct a workforce analysis to inventory existing employee skills and abilities and develop strategies for addressing any SMS-related gaps identified. 5. To maximize the positive impact of SMS implementation on aviation safety, consider strategies to address airports’ concerns that may negatively affect data collection and data sharing, including asking Congress to provide additional protections for SMS data collected by public entities. We provided the Department of Transportation and NTSB with a draft of Agency Comments this report for review and comment. DOT and NTSB officials provided technical comments, which we incorporated as appropriate and DOT agreed to consider the recommendations. In addition, DOT officials stated there is a need for FAA to have a common hazard-tracking system. FAA has taken initial steps towards standardization by publishing FAA Order 8040.4A, Safety Risk Management Policy, which identifies terms and definitions used for safety risk management. DOT also reinforced its dedication to the success of SMS and noted its continued efforts to improve its implementation plans with a measured, structured approach to implementation. Page 33 GAO-12-898 Safety Management Systems We are sending copies of this report to the appropriate congressional committees, DOT, NTSB, and interested parties, and others. In addition, the report will be available at no charge on the GAO Web site at http://www.gao.gov. If you or your staff members have any questions about this report, please contact me on (202) 512-2834 or at email@example.com. Contact points for our Office of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix II. Gerald L. Dillingham, Ph.D. Director Physical Infrastructure Issues Page 34 GAO-12-898 Safety Management Systems Appendix I: Objectives, Scope, and Appendix I: Objectives, Scope, and Methodology Methodology Our objective was to assess the Federal Aviation Administration’s (FAA) implementation of Safety Management Systems (SMS) and provide information on potential implementation challenges. To do so, we addressed the following questions: (1) What is the status of FAA’s implementation of SMS? (2) To what extent have FAA’s SMS efforts been consistent with key practices for successful planning and implementation of a new program? (3) What challenges does FAA face in implementing SMS? To perform our review, we focused primarily on FAA’s implementation of SMS for its business lines as well as its preliminary efforts to require and oversee SMS implementation by industry. We conducted background research to identify literature related to SMS in aviation, and any challenges that agencies might face when implementing SMS. We also attended parts of a safety risk management panel on runway status lights conducted by FAA’s Air Traffic Organization (ATO) at Seattle-Tacoma International Airport in March 2012 as a means of learning more about SMS and related processes. During the data collection and drafting phases of this report, FAA was in the midst of rulemaking efforts to require SMS of Part 121 air carriers and Part 139 airports, so we did not comment on any draft or proposed regulatory guidance. To determine the status of FAA’s implementation of SMS, we reviewed FAA’s SMS orders and pilot project guidance, implementation plans, and Notices of Proposed Rulemaking for Part 121 air carriers and Part 139 airports. We also reviewed international and FAA guidance on SMS issued by the International Civil Aviation Organization (ICAO) and the Joint Planning and Development Office (JPDO), respectively, and National Transportation Safety Board (NTSB) recommendations to FAA related to SMS. Finally, we interviewed FAA SMS program managers across FAA business lines and offices; industry experts we identified based on their knowledge and experience in industry, recommendations from aviation industry officials, and a search of SMS literature; and ICAO and NTSB officials. To assess the extent to which FAA’s efforts have been consistent with key practices, we reviewed our reports and other literature on successful project planning and implementation, particularly for large-scale transformative projects, and condensed the resulting list to eliminate Page 35 GAO-12-898 Safety Management Systems Appendix I: Objectives, Scope, and Methodology duplication and overlap. To do this, we reviewed previous GAO reports that highlighted practices associated with successful planning and implementation of a new program. 1 We removed or consolidated any duplicate items across the reports to create a single list of 10 criteria. We then identified FAA’s actions related to these practices by reviewing FAA guidance and agency documentation such as its SMS implementation plans, conducting interviews with FAA officials across its business lines, and using that information to assess the extent to which FAA had addressed each practice. We determined whether each key practice was addressed, partially addressed, or not addressed by using criteria developed for prior GAO reports. As such, we considered a practice “addressed” if FAA had instituted the practice; “partially addressed” if FAA had shown some progress toward instituting, or started but not completed the practice; and “not addressed” if FAA had made minimal or no progress toward instituting the practice. The team made these coding decisions together, with two analysts making initial judgments and team management reviewing and confirming them. To identify challenges FAA faces in implementing SMS, we reviewed our prior work on long-standing FAA challenges, such as those related to training and data, and interviewed aviation industry experts and FAA officials mentioned above. We reviewed prior GAO work on performance measurement and workforce analysis, Department of Transportation Inspector General reports and NTSB recommendations related to SMS. To obtain industry views on challenges, we interviewed officials from selected airports and air carriers, industry associations representing airports, air carriers, and pilots, and individuals with SMS experience described above. We also reviewed and analyzed documents, including language in the FAA Modernization and Reform Act of 2012 related to data protection, and associated scholarly work. To supplement comments received from the individuals we interviewed, we also reviewed comments made by aviation stakeholders on the two Notices of Proposed Rulemaking related to SMS. 1 For examples, see GAO, Results-Oriented Cultures: Implementation Steps to Assist Mergers and Organizational Transformations, GAO-03-669 (Washington, D.C.: July 2, 2003) and GAO, Managing for Results: GPRA Modernization Act Provides Important Opportunities to Address Government Challenges, GAO-11-617T (Washington, D.C.: May 10, 2011). Page 36 GAO-12-898 Safety Management Systems Appendix I: Objectives, Scope, and Methodology To obtain industry views on both SMS implementation practices and associated challenges, we interviewed officials from selected airports and air carriers, which we selected for diversity in size, location, participation in FAA SMS pilot projects, and submission of comments on FAA’s two Notices of Proposed Rulemaking related to SMS. (See table 1 for a list of selected airports.) Table 1: Airport Interviews Airport State Concord Regional North Carolina Dallas/Fort Worth International Texas King County International Washington Los Angeles International California Manchester-Boston Regional New Hampshire Pittsburgh International Pennsylvania Sacramento International California Seattle-Tacoma International Washington Sloulin Field International North Dakota South Bend Regional Indiana Source: GAO. We also interviewed officials from six air carriers: Delta, 2 GoJet, United, 3 Pinnacle, Southwest, and US Airways. Finally, we interviewed officials with SMS knowledge and expertise, including experts from the Flight Safety Foundation, Embry-Riddle Aeronautical University, John A. Volpe National Transportation Systems Center, and MITRE Corporation. 2 Northwest Airlines merged with Delta Air Lines in 2008. 3 Continental Airlines merged with United Airlines in 2010. Page 37 GAO-12-898 Safety Management Systems Appendix II: GAO Contact and Staff Appendix II: GAO Contact and Staff Acknowledgments Acknowledgments Gerald L. Dillingham, Ph.D., (202) 512-2834, or firstname.lastname@example.org GAO Contact In addition to the contact named above, Heather MacLeod (Assistant Staff Director); Elizabeth Curda; Leia Dickerson; Sarah Farkas; David Hooper; Acknowledgments Delwen Jones; Brooke Leary; Josh Ormond; Larry Thomas; and Elizabeth Wood made key contributions to this report. (540230) Page 38 GAO-12-898 Safety Management Systems GAO’s Mission The Government Accountability Office, the audit, evaluation, and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO’s commitment to good government is reflected in its core values of accountability, integrity, and reliability. The fastest and easiest way to obtain copies of GAO documents at no Obtaining Copies of cost is through GAO’s website (www.gao.gov). Each weekday afternoon, GAO Reports and GAO posts on its website newly released reports, testimony, and correspondence. To have GAO e-mail you a list of newly posted products, Testimony go to www.gao.gov and select “E-mail Updates.” Order by Phone The price of each GAO publication reflects GAO’s actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO’s website, http://www.gao.gov/ordering.htm. Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537. Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information. Connect with GAO on Facebook, Flickr, Twitter, and YouTube. Connect with GAO Subscribe to our RSS Feeds or E-mail Updates. Listen to our Podcasts. Visit GAO on the web at www.gao.gov. Contact: To Report Fraud, Waste, and Abuse in Website: www.gao.gov/fraudnet/fraudnet.htm E-mail: email@example.com Federal Programs Automated answering system: (800) 424-5454 or (202) 512-7470 Katherine Siggerud, Managing Director, firstname.lastname@example.org, (202) 512- Congressional 4400, U.S. Government Accountability Office, 441 G Street NW, Room Relations 7125, Washington, DC 20548 Chuck Young, Managing Director, email@example.com, (202) 512-4800 Public Affairs U.S. Government Accountability Office, 441 G Street NW, Room 7149 Washington, DC 20548 Please Print on Recycled Paper.
Aviation Safety: Additional FAA Efforts Could Enhance Safety Risk Management
Published by the Government Accountability Office on 2012-09-12.
Below is a raw (and likely hideous) rendition of the original report. (PDF)