NASA National Aeronautics and Space Administration Office of Inspector General Office of Audits NASA’S RESPONSE TO SPACEX’S JUNE 2015 LAUNCH FAILURE: IMPACTS ON COMMERCIAL RESUPPLY OF THE INTERNATIONAL SPACE STATION June 28, 2016 Report No. IG-16-025 Office of Inspector General To report, fraud, waste, abuse, or mismanagement, contact the NASA OIG Hotline at 800-424-9183 or 800-535-8134 (TDD) or visit https://oig.nasa.gov/hotline.html. You can also write to NASA Inspector General, P.O. Box 23089, L’Enfant Plaza Station, Washington, D.C. 20026. The identity of each writer and caller can be kept confidential, upon request, to the extent permitted by law. To suggest ideas for or to request future audits contact the Assistant Inspector General for Audits at https://oig.nasa.gov/aboutAll.html. RESULTS IN BRIEF NASA’s Response to SpaceX’s June 2015 Launch Failure: Impacts on Commercial Resupply of the International Space Station NASA Office of Inspector General Office of Audits June 28, 2016 IG-16-025 (A-15-013-00) WHY WE PERFORMED THIS AUDIT On June 28, 2015, just 2 minutes after liftoff, Space Exploration Technologies Corporation’s (SpaceX) seventh cargo resupply mission (SPX-7) to the International Space Station (ISS or Station) failed, destroying $118 million of NASA cargo, including an International Docking Adapter (Adapter) the Agency planned to use when it begins flying astronauts to the Station on commercial vehicles. In the aftermath of the failure, SpaceX suspended resupply missions pending completion of an investigation into its cause, relicensing of its launch vehicle by the Federal Aviation Administration (FAA), and acceptance by NASA of the company’s corrective actions. SPX-7 was the second commercial resupply mission failure in an 8-month period. In October 2014, Orbital ATK’s (Orbital) third resupply mission crashed near the launch pad, destroying the company’s rocket and capsule as well as $51 million of NASA cargo.1 SpaceX and Orbital have fixed-price cargo resupply contracts worth a maximum value of $3.1 billion each. In light of these events and because, to date, SpaceX and Orbital are the only two U.S. companies transporting supplies to the Station, we examined NASA’s response to the SpaceX failure and its impact on commercial resupply of the ISS. As part of this review, we assessed the technical and operational risks of SpaceX’s plans for resuming resupply missions, NASA’s efforts to reduce the financial and other risks associated with its contract with SpaceX, and the procedures for investigating the cause of the failure. We also reviewed relevant policies, regulations, and procedures; interviewed NASA, SpaceX, and other officials; and conducted site visits at SpaceX headquarters and the Johnson Space Center. WHAT WE FOUND Due to the loss of SPX-7 and the shift of SpaceX’s eighth resupply mission into 2016, approximately 3.48 metric tons (3,480 kilograms [kg]) of pressurized cargo scheduled for delivery in fiscal year (FY) 2015 did not arrive on the Station. NASA was able to absorb this loss because increased packing efficiencies and high cargo densities enabled transport of an additional 746 kg of upmass on two other SpaceX cargo missions and a Japanese cargo flight. In addition, the Russian space agency carried an additional 100 kg of pressurized upmass for NASA over six different flights. These measures reduced the total upmass shortfall from 3.48 metric tons to 2.63 metric tons (2,630 kg). Furthermore, the SpaceX and Orbital mission failures have led to a compressed launch schedule in FYs 2016 and 2017, with 11 cargo resupply missions, 7 Russian cargo missions, and 1 Japanese cargo mission now scheduled to arrive at the Station. In mid-2014, NASA astronauts were spending as much as 44 hours a week on research and related activities. While program officials stated that the number of research hours will not fall below the 35-hour/week minimum, the total time devoted to research may decrease from 2014 levels as astronauts take time to receive, unpack, and repack all of these vehicles. 1 In a September 2015 report, we examined NASA’s response to the Orbital’s launch failure. NASA Office of Inspector General, “NASA’s Response to Orbital’s October 2014 Launch Failure: Impacts on Commercial Resupply of the International Space Station” (September 17, 2015, IG-15-023). The most significant item lost during the SPX-7 mission was the first of two Docking Adapters necessary to support upcoming commercial crew missions. Although NASA had planned to have two Adapters installed on the Station before the first commercial crew demonstration mission scheduled for May 2017, it is now likely there will be only one installed in time for these missions. Having only one Adapter means that a commercial crew vehicle will not be able to dock with the ISS if technical issues arise with the single available docking port. ISS Program officials stated that they plan to have the replacement Adapter installed before regular commercial crew rotations begin. We found NASA is effectively managing its commercial resupply contract with SpaceX to reduce cost and financial risk. The Agency has taken advantage of multiple mission pricing discounts and negotiated equitable adjustments of significant value to the Agency. In addition, following the SPX-7 failure NASA negotiated significant consideration in the form of Adapter hardware, integration services, manifest flexibility, and discounted mission prices for the SPX-16 through SPX-20 resupply missions. However, we also found that for the first seven cargo missions NASA did not fully utilize the unpressurized cargo space available in the Dragon 1 capsule’s trunk, averaging 423 kg for SPX-3 through SPX-7 even though the trunk is capable of carrying more. The ISS Program noted that unpressurized payloads depend on manifest priority, payload availability, and mission risk, and acknowledged it struggled to fully utilize this space on early missions, but as of June 2016 the Agency’s cargo manifests show full trunks on all future SpaceX cargo resupply missions. Finally, the ISS Program adopted a tailored risk management approach for commercial cargo launches that deviated from existing procedures for evaluating launch risks. In practice, NASA has treated all commercial resupply missions as the lowest level risk classification irrespective of a mission’s value and relies primarily on its commercial partners (SpaceX and Orbital) to evaluate and mitigate launch risks. As a result, risk mitigation procedures are not consistently employed and the subjective launch ratings the Agency uses provide insufficient information to NASA management concerning actual launch risks. In addition, NASA does not have an official, coordinated, and consistent mishap investigation policy for commercial resupply launches, which could affect its ability to determine the root cause of a launch failure and implement corrective actions. WHAT WE RECOMMEND In order to maintain the efficacy of the ISS and ensure delivery of cargo in a timely and affordable manner, we recommend the Associate Administrator for Human Exploration and Operations ensure the ISS Program (1) incorporates the risk of limited availability of the Adapter into risk management processes; (2) continues to refine the unpressurized upmass manifesting process and considers preparing alternative unpressurized upmass payloads in the event scheduled payloads cannot be launched; (3) quantifies overall mission risk ratings and communicates the risks for upcoming launches early and in coordination with varying levels of engineering and management; and (4) reviews all investigation authorities and plans during commercial launches with NASA payloads to ensure they are standardized. To clarify the division of roles and responsibilities in the event of a mission failure, we recommend the Office of Safety and Mission Assurance, in conjunction with ISS Program officials, (5) improve coordination with other Federal agencies involved in commercial space and (6) update NASA procedures to include commercial space launches with NASA payloads in official mishap policies. In response to a draft of our report, the Associate Administrator concurred or partially concurred with five of our recommendations and described corrective actions the Agency has taken or will take to address them. Those recommendations are resolved and will be closed upon completion and verification of the proposed corrective actions. NASA did not concur with our recommendation to quantify overall mission risk ratings and communicate the risks for upcoming launches. Therefore, the recommendation is unresolved pending further discussion with Agency officials. For more information on the NASA Office of Inspector General and to view this and other reports visit https://oig.nasa.gov/. TABLE OF CONTENTS Introduction .................................................................................................................................................. 1 Background ............................................................................................................................................... 1 Failure Contributed to Cargo Shortfall and More Flights in Fiscal Years 2016 and 2017 and Resulted in Loss of Docking Adapter ......................................................................................................... 11 Upmass Shortfall and Additional Cargo Missions in FYs 2016 and 2017 ................................................ 11 Consequences of Loss of International Docking Adapter ....................................................................... 13 NASA Benefited from Discounted Mission Pricing and Negotiated Valuable Consideration, But Has Not Fully Utilized SpaceX Cargo Capacity .................................................................................... 15 NASA Negotiated Favorable Pricing for Initial Flights and Contract Extensions ..................................... 15 NASA Effectively Negotiated Contract Modifications and Received Good Value for Adjustments ...................................................................................................................................... 16 NASA Did Not Fully Utilize SpaceX’s Upmass Capability ......................................................................... 17 NASA Could Improve Risk Management for Commercial Cargo Resupply Missions ............................... 19 ISS Program Deviated from Existing Agency Risk Processes ................................................................... 19 NASA Primarily Relies on SpaceX and Orbital to Evaluate and Mitigate Launch Risk ............................ 20 Orb-3 Review Team Expressed Concerns about Risk Management ....................................................... 21 Standardization and Independence Requirements Could Improve CRS-1 Mishap Investigations ................................................................................................................................ 23 NASA Could Improve Investigation Policies and Coordination for CRS-1 Launches ............................... 23 Lack of Independence Could Inhibit Contractor-Led Investigations ....................................................... 25 Conclusion .................................................................................................................................................. 27 Recommendations, Management’s Response, and Our Evaluation ........................................................ 28 Appendix A: Scope and Methodology ...................................................................................................... 31 Appendix B: NASA Payload and Launch Vehicle Risk Classifications....................................................... 34 Appendix C: USAF Criteria for Launch and Payload Risk .................................................................... 36 Appendix D: CRS-1 Accident Plans and Investigations ............................................................................. 39 Appendix E: Management’s Comments ................................................................................................... 42 Appendix F: Report Distribution ............................................................................................................... 48 NASA Office of Inspector General IG-16-025 i Acronyms CRS Commercial Resupply Services EVA Extravehicular Activity FAA Federal Aviation Administration FAR Federal Acquisition Regulation FRAM Flight Releasable Attachment Mechanism FY Fiscal Year ISS International Space Station kg Kilogram LSP Launch Services Program NPD NASA Policy Directive NPR NASA Procedural Requirements NTSB National Transportation Safety Board Orb Orbital Commercial Resupply Services Mission OIG Office of Inspector General RAC Risk Assessment Code SPX SpaceX Commercial Resupply Services Mission USAF United States Air Force NASA Office of Inspector General IG-16-025 ii INTRODUCTION On June 28, 2015, the Space Exploration Technologies Corporation (SpaceX) – under contract with NASA – launched its seventh commercial resupply services mission (SPX-7) to the International Space Station (ISS or Station) from Florida’s Cape Canaveral Air Force Station. Just over 2 minutes after liftoff, launch data indicated a problem with the second stage of SpaceX’s Falcon 9 rocket and the vehicle broke up in flight, destroying the rocket along with the Dragon capsule and $118 million worth of NASA cargo it was carrying. In the aftermath of the failure, SpaceX suspended further resupply missions pending completion of an investigation into the accident’s cause, relicensing of its launch vehicle by the Federal Aviation Administration (FAA), and acceptance by NASA of the company’s corrective actions.1 SPX-7 was the second commercial resupply mission failure in an 8-month period. In late October 2014, Orbital ATK’s (Orbital) third commercial resupply services mission (Orb-3) failed during lift-off, causing the vehicle to crash near the launch pad and destroying Orbital’s rocket and capsule as well as $51 million of NASA cargo.2 We issued a report in September 2015 that examined NASA’s response to the Orb-3 launch failure.3 In light of these events and because, to date, SpaceX and Orbital are the only two U.S. companies transporting supplies to the Station, we examined NASA’s response to the SpaceX failure and its impact on commercial resupply of the ISS. As part of this review, we assessed the technical and operational risks of SpaceX’s plans for resuming resupply missions, NASA’s efforts to reduce the financial and other risks associated with its contract with SpaceX, and the procedures for investigating the cause of the failure. See Appendix A for details of our scope and methodology. Background Between 2006 and 2008, NASA entered into a series of funded Space Act Agreements with SpaceX and Orbital to stimulate development by U.S. corporations of transportation systems capable of providing cargo delivery services to low Earth orbit, including the ISS.4 In addition to receiving more than $700 million total from NASA, SpaceX and Orbital committed their own resources to this effort, ultimately contributing more than 50 percent of the development costs of their respective spaceflight systems. 1 NASA cargo resupply launches are licensed through the FAA pursuant to the Commercial Space Launch Act, as amended and recodified as “Commercial Space Launch Activities” at 51 U.S.C §§ 50901–923. 2 In February 2015, Orbital Sciences Corporation merged with Alliant Techsystems, Incorporated to form Orbital ATK. For ease of reference, we refer to the corporation as Orbital in this report. 3 NASA Office of Inspector General, “NASA’s Response to Orbital’s October 2014 Launch Failure: Impacts on Commercial Resupply of the International Space Station” (September 17, 2015, IG-15-023). 4 In funded Space Act Agreements, NASA transfers funds to a partner to undertake activities consistent with NASA missions. Pursuant to Agency policy, NASA may only use funded Space Act Agreements when the Agency cannot accomplish its objectives using a more traditional vehicle such as a contract, grant, or cooperative agreement. NASA Office of Inspector General IG-16-025 1 Commercial Resupply Services Contracts In 2008, while development efforts were still underway, NASA awarded fixed-price contracts worth a maximum value of $3.1 billion each to SpaceX and Orbital under its first Commercial Resupply Services (CRS-1) contract. NASA selected two companies, in part to ensure redundancy if one was unable to perform, and awarded the contracts in accordance with Federal Acquisition Regulation (FAR) Part 12.5 Shortly after contract award, NASA issued task orders to SpaceX and Orbital valued at approximately $1.6 billion and $1.9 billion, respectively, for a series of cargo missions to the ISS. The contracted services included delivery of supplies and equipment (upmass) to the Station and, depending on the mission, return of equipment and experiments or disposal of waste (downmass) to Earth.6 Key Features of CRS-1 Contracts NASA implements the CRS-1 contracts through a series of task orders and work plans detailing specific objectives for each resupply mission and identifying milestones each company must meet to secure payment, criteria by which the Agency determines whether a particular milestone has been achieved, launch dates, and payment allocations for milestone completion. Once NASA and the companies agree a particular milestone has been accomplished, NASA pays the company a predetermined amount associated with the milestone. The first milestone for both SpaceX and Orbital is authorization from the NASA Contracting Officer to begin work on a mission, known as “authorization to proceed,” and the final two milestones are launch and delivery of cargo to the ISS. The companies are not paid for milestones they do not achieve. For example, because of the SPX-7 failure, SpaceX forfeited the final 30 percent in milestone payments. Similarly, since Orb-3 did not deliver cargo to the ISS, NASA did not pay Orbital the final 20 percent for the delivery milestone.7 Originally, NASA guaranteed that it would purchase a minimum of 20 metric tons (approximately 44,000 pounds) of upmass from each company between 2010 and 2015.8 SpaceX, whose capsule returns to Earth, agreed to carry at least three metric tons of downmass back to Earth, while Orbital, whose capsule disintegrates upon atmospheric reentry, agreed to provide trash disposal services matching its upmass capability. In 2009, NASA issued the first in a series of task orders to detail the 5 FAR Part 12, “Acquisition of Commercial Items,” implements the Federal Government’s preference for the acquisition of commercial items as expressed in 41 U.S.C. §§ 1906, 1907, and 3307 and 10 U.S.C. §§ 2375–2377, by establishing acquisition policies more closely resembling those of the commercial marketplace and encouraging the acquisition of commercial items and components. 6 In addition to the SpaceX and Orbital missions, NASA barters with the Japan Aerospace Exploration Agency for cargo transportation on Japan’s H-II Transfer Vehicle and has placed small amounts of upmass on the Russian Space Agency’s Soyuz capsule and Progress cargo vehicle. In the past, NASA also sent cargo to the ISS on the European Space Agency’s Automated Transfer Vehicle, which made its final delivery in July 2014. 7 Although milestones have some commonality between the two providers, the criteria for completing milestones and amounts of payment can vary. 8 The purpose of the guarantee was to ensure a minimum payment to each company sufficient to create a business case for providing cargo services. The guarantee means that as long as it provides a vehicle capable of carrying the agreed upon weight for each mission, the company meets its contractual requirements regardless of the amount of cargo NASA actually presents for transport. NASA Office of Inspector General IG-16-025 2 expected upmass and cost of each mission. The initial contracts required SpaceX to transport 39.7 metric tons over 12 missions and Orbital 19.3 metric tons over 8 missions.9 These values were reduced in subsequent discussions between NASA and the companies to 35.4 metric tons for SpaceX and 18.6 metric tons for Orbital in exchange for the companies providing additional cargo and waste disposal capabilities. The CRS-1 contracts provide two options for pricing resupply missions: (1) using tables that set gradually increasing prices depending on the year of launch and provide a discount when multiple missions are flown in a single year (mission pricing) or (2) by kilograms (kg) of cargo (mass pricing).10 The SpaceX contract features two provisions relating to mass pricing. First, all per kilogram pricing assumes cargo mass capacity is fully utilized (3,310 kg upmass and 3,310 kg downmass). Second, the cargo load is variable and could be volume limited rather than mass limited. This means that if the manifested cargo for a particular mission reaches the volume limit before the maximum weight, NASA nevertheless must pay the company the full mission price. With the exception of the companies’ demonstration flights, NASA has used mission pricing for all CRS-1 missions. The CRS-1 contracts place much of the risk associated with an unsuccessful mission on NASA. However, this is not unusual for Government contracts relating to space operations given the associated expense and risks, and the limited number of capable contractors. Due to the relationship between risk and price, shifting more risk to the contractor would likely increase contract price. To this end, the CRS-1 contracts do not require SpaceX or Orbital to re-fly failed missions or carry upmass from a failed mission on future flights, nor do they make the companies liable for any cargo destroyed as a result of a launch failure or other anomaly. While, as previously noted, if SpaceX or Orbital fail to deliver cargo to the Station the companies forfeit any payment tied to the associated milestones, NASA is not entitled to recover previous milestone payments associated with the launch. Furthermore, the Agency can only recover milestone payments it has made toward missions not yet flown if it terminates the contract for cause – known as “termination for cause.”11 Contract Changes The NASA contracting officer may make changes to the general scope of the CRS-1 contracts in accordance with FAR 52.243-1.12 This clause gives the contracting officer authority to revise the original terms and conditions, and incorporate modifications reflecting equitable adjustments agreed to between the Government and the contractor. Specifically, it provides that the contracting officer shall make an equitable adjustment in the contract price or the delivery schedule, or both, and modify the contract when contract changes cause an increase or decrease in the cost of or the time required for performance of any part of the work under the contract. Equitable adjustments may result from either a Government-desired change or to incorporate consideration to the Government when a contractor is not meeting requirements. 9 NASA originally ordered 12 flights from SpaceX (1 each in 2010 and 2011, 2 in 2012, 3 each in 2013 and 2014, and 2 in 2015) and 8 flights from Orbital (1 each in 2011 and 2012, and 2 each in 2013 through 2015). 10 The pricing tables contain separate figures for pressurized and unpressurized capsules and returned or disposed downmass. 11 Under a “termination for cause” scenario, the Government may terminate all or a portion of a commercial contract if the contractor fails to comply with contract terms or cannot provide the Government with adequate assurances of future performance. We inquired with another Federal agency that procures launch services to insert payloads into orbit and were informed that the agency typically structures its contracts similarly to the CRS-1 contracts with a relatively small final payment tied to successful launch and the contractor retaining prior milestone payments in the event of a mishap. 12 FAR § 52.243-1, “Changes—Fixed-Price” (August 1987). NASA Office of Inspector General IG-16-025 3 Second Round of Commercial Cargo Resupply Contracts In January 2016, NASA awarded the second round of commercial resupply services (CRS-2) contracts to Orbital, SpaceX, and the Sierra Nevada Corporation (Sierra Nevada).13 The maximum combined potential value of the CRS-2 contracts is $14 billion with a period of performance from 2016 through 2024.14 NASA is expected to order a minimum of six missions from each provider at fixed prices with specified cargo amounts and performance dates based on the Station’s needs. SpaceX and Orbital will continue to fly capsule designs similar to those used for their CRS-1 contracts with some modifications, while Sierra Nevada will use its Dream Chaser.15 Table 1 outlines the capabilities expected from each service provider. Table 1: Cargo Delivery Services Cygnus Dragon Dream Chaser Company Orbital SpaceX Sierra Nevada Launch Vehicle Antares or Atlas V Falcon 9 Atlas V Upmass 3,200–3,500 kg 3,310 kg 5,500 kga Capabilities Downmass Disposal only Disposal or return to Earth Disposal or return to Earth Capabilities Number of At least 6 At least 6 At least 6 Missions Source: NASA Office of Inspector General summary of contract requirements and vehicle capabilities. a Upmass capability for Dream Chaser is based on company projections at the time of publication and has not been demonstrated. 13 SpaceX and The Boeing Company (Boeing) also hold contracts with NASA to transport astronauts to and from the ISS beginning as early as 2017. 14 The first CRS-2 missions are expected in 2019. 15 The Dream Chaser is a winged vehicle that resembles a mini Space Shuttle and, like the Shuttle, launches aboard a rocket but glides back to Earth to land on a runway. NASA Office of Inspector General IG-16-025 4 As a result of prior NASA Office of Inspector General (OIG) recommendations and lessons learned from the CRS-1 experience, NASA changed the CRS-2 contracts in several respects.16 First, the CRS-2 contracts provide NASA with the flexibility to order flight support equipment for a given mission when needed rather than tying such orders to the authorization to proceed milestone for the entire mission as was the case for CRS-1. This change allows the Agency to reduce the amount of funding it provides upfront when it orders a mission. Second, NASA linked payments for Station integration milestones to other mission milestones, which requires the contractors to demonstrate their vehicles can safely approach the ISS before they receive associated payments. NASA also created a requirements change line item that enables the Agency to evaluate the need for changes in the contract on an annual basis and keep the contract current with ISS needs.17 Finally, although the CRS-1 contract included a vague reference to the companies obtaining Agency approval to fly non-NASA payloads, in CRS-2 NASA included a specific clause that defines the rules of engagement for flying non-NASA payloads or performing other contractor objectives on CRS-2 flights. These changes are meant to provide NASA with more flexibility in managing the commercial resupply contracts. In addition, NASA added provisions to the CRS-2 contract aimed at improving its insight into contractor operations. For example, for the CRS-1 contract NASA officials had limited access to important information on Orbital’s rocket engines which made risk assessment difficult. NASA has revised this clause in the CRS-2 contract to clarify that in some instances contractors may need to execute third-party data agreements to allow NASA to review subcontractor designs, processes, and parts to enable a more thorough risk assessment. NASA also added language requiring contractors to show how design changes in the cargo capsule or launch vehicle affect performance and risk margins. SpaceX Capabilities SpaceX has two variations of its Dragon capsule: one for cargo delivery (Dragon 1) and the other for crew transportation (Dragon 2). The company is designing Dragon 2 to transport up to seven crew members, with an abort system in case of emergency and the capability of either propulsive ground landing or parachute-to-water landing. The first Dragon 2 demonstration mission to the ISS – scheduled for May 2017 – will not carry a crew. The second demonstration mission, which will carry a crew, is scheduled for August 2017. See Table 2 for a comparison of the Dragon 1 and Dragon 2 capabilities. 16 NASA OIG, IG-15-023. 17 A contract line item typically specifies the product or service being procured and the negotiated price for that item. NASA Office of Inspector General IG-16-025 5 Table 2: Dragon 1 and Dragon 2 Capabilities Berthing/ Demonstration Purpose Payload Docking Missions 3,310 kg cargo Cargo December 2010 Dragon 1 Pressurized (11 m3) or Common Berthing Mechanism transportation and May 2012 unpressurized (14 m3) International Docking Adapter/ Crew 7 crew members or 4-5 May 2017 and Dragon 2 International Docking System transportationa crew plus cargo August 2017b Standard Source: NASA OIG analysis of ISS Program and SpaceX information. Note: m3 denotes cubic meters a A variant of the Dragon 2 is being developed to transport cargo under the CRS-2 contract. b Scheduled launch dates as of June 2016. Under the CRS-1 contract, Dragon 1 was the first commercial spacecraft to berth with the ISS, executing six successful missions to the Station prior to SPX-7. As shown in Figure 1, the Dragon 1 capsule consists of an 11 m3 (about 388 cubic feet) pressurized cargo module and a 14 m3 (about 494 cubic feet) unpressurized trunk.18 Dragon 1 is currently the only commercial cargo vehicle capable of returning cargo such as science experiments from the ISS to Earth. Figure 1: Dragon 1 Diagram Source: NASA OIG presentation of SpaceX information. 18 Pressurized cargo is kept at an Earth-like atmospheric pressure environment. In contrast, unpressurized cargo is carried externally to the cargo vehicle and can be exposed to the space environment. NASA Office of Inspector General IG-16-025 6 With the exception of SpaceX’s first two missions (SPX-1 and SPX-2), which delivered 450 kg and 865 kg to the ISS, respectively, NASA has generally loaded Dragon 1’s pressurized module to its volumetric limit.19 However, the amount of upmass stored in the module and trunk has varied by mission based on NASA’s needs and the volume and density of particular cargo. Total pressurized and unpressurized upmass for the next five missions (SPX-3 through SPX-7) ranged from 2,024 kg to 2,478 kg. During these missions, Dragon 1 was loaded with an average of 1,847 kg of pressurized cargo and an average of 423 kg of unpressurized cargo. To date, SpaceX has successfully transported 13,446 kg of upmass to the ISS and 8,803 kg of downmass back to Earth, averaging 1,467 kg of downmass per mission.20 SpaceX expects to transport more upmass per flight in its upcoming flights by substantially increasing the amount of unpressurized cargo the Dragon 1 carries. For example, despite averaging only 423 kg of unpressurized cargo per mission on SPX-3 through SPX-7, the company transported more than 1,500 kg of unpressurized payload on SPX-8 and projects it will carry approximately the same amount on SPX- 11.21 As a result of the increase in unpressurized payload SPX-8 nearly met its maximum upmass capacity of 3,310 kg. SPX-7 Cargo SPX-7 was carrying more than 1,900 kg in its pressurized module consisting of the following items: Crew supplies, including food and other provisions (690 kg). Utilization equipment, including science experiments and supporting equipment for the Canadian Space Agency, European Space Agency, Japan Aerospace Exploration Agency, and NASA (573 kg). Vehicle hardware, including tanks and filter inserts necessary for the Station’s Environmental Control and Life Support System (Life Support System) (462 kg). Extravehicular activity (EVA) equipment, including an astronaut spacesuit (Short Extravehicular Mobility Unit) (167 kg).22 Computer resources, including a projection screen, laptop, and various power modules (36 kg). In addition, SPX-7 was carrying 526 kg of unpressurized cargo, including the first of two International Docking Adapters (Adapter) designed to update the ISS’s docking system so that the crew vehicles under development by SpaceX and Boeing will be able to dock with the Station. The Adapter cost NASA $32.4 million to develop. The second Adapter will be flown on SPX-9, which is scheduled for launch in July 2016. 19 The first two missions carried smaller loads because the empty cargo vehicles were heavier than expected and the Falcon 9 rocket did not meet its planned lift capability. SpaceX has since addressed both of these issues with an upgrade to its Falcon 9 rocket. 20 The downmass figure does not include the SPX-8 mission, which had not yet returned to Earth at the time these figures were calculated. 21 Unpressurized cargo will be a bit lower for SPX-9 and SPX-10 (550 kg and 977 kg, respectively) due to the lower weights of the manifested payloads. 22 EVA (spacewalk) is performed by astronauts outside the Station while on orbit. NASA Office of Inspector General IG-16-025 7 SpaceX’s Return to Flight Plan Following the SPX-7 failure, SpaceX recovered parts of the Falcon 9 rocket and, through telemetry analysis and other testing, determined the most probable cause for the mishap was a strut assembly failure in the rocket’s second stage. Specifically, the failed strut assembly released a helium tank inside the liquid oxygen tank, causing a breach in the oxygen tank’s dome and the release of gas that in turn disabled the avionics and caused release of the Dragon 1 capsule and break-up of the launch vehicle. SpaceX completed an extensive analysis of the SPX-7 failure, consulted with NASA and the United States Air Force (USAF) regarding their analysis, and provided a mishap report and Return to Flight Plan to the FAA and NASA in November 2015. The company’s post-mishap testing of strut parts from the same purchase order as those used on SPX-7 found material flaws due to casting defects, “out of specification” materials, and improper heat treatment.23 NASA’s Launch Services Program (LSP) conducted a separate, independent review of the failure, briefing its results to senior NASA leadership on December 18, 2015.24 LSP did not identify a single probable cause for the launch failure, instead listing several “credible causes.” In addition to the material defects in the strut assembly SpaceX found during its testing, LSP pointed to manufacturing damage or improper installation of the assembly into the rocket as possible initiators of the failure. LSP also highlighted improper material selection and such practices as individuals standing on flight hardware during the assembly process, as possible contributing factors.25 SpaceX has taken action to correct the deficiencies that led to the failed strut assembly and to address NASA’s concerns by conducting inspections, replacing suspect parts, and conducting additional testing. The company also reviewed the certifications of all spaceflight hardware and altered its quality control processes to better align with NASA technical standards. In order to track completion of its corrective actions, SpaceX is updating its process for identifying and resolving work-related tasks, which allows for improved auditing, prioritizing, and tracking of fracturable hardware. To administer its updated quality control process, SpaceX has reorganized into three teams called “Design Reliability,” “Build Reliability,” and “Flight Reliability.” Besides monitoring corrective actions taken as a result of the SPX-7 failure, these teams are tracking the significant upgrades SpaceX has made to the Falcon 9 launch system for future launches, including increased thrust capability with a new fuel mixture and corrective actions on software implementation plans, which are both rated as low risks by the ISS Program. CRS-1 Contract Modifications After the SPX-7 failure and through a series of negotiations, NASA modified SpaceX’s CRS-1 contract in December 2015 to add five additional flights – SPX-16 through SPX-20 – at discounted prices, as well as 23 A casting defect is an irregularity that occurs when molten metal is poured into a mold and cooled. An “out of specification” material has a technical attribute (e.g., chemical composition, mechanical property) outside of the prescribed values for the type of metal specified for a particular use. Heat treatment at accurate temperatures strengthens metal parts while improper heat treatment can cause deviations or weaknesses. 24 LSP purchases commercial launch services for NASA customers, including missions of the Agency’s Science Mission Directorate. LSP had a contract with SpaceX to use the Falcon 9 to deliver a science mission payload. 25 In February 2016, the NASA Administrator and the Associate Administrator for the Human Exploration and Operations Mission Directorate sent a letter to SpaceX expressing concerns about the company’s systems engineering and management practices, hardware installation and repair methods, and telemetry systems based on LSP’s review of the failure. NASA Office of Inspector General IG-16-025 8 hardware, integration activities, and manifest flexibility at no cost to the Agency.26 In addition, the revised contract provides that SpaceX will satisfy NASA’s remaining upmass requirements, and the company plans to fly heavier payloads on future missions. The heavier payloads are possible because the ISS Program has resolved past difficulties in maximizing the use of the unpressurized section of the cargo capsule. With these improvements, SpaceX officials expect SPX-11 through SPX-15 to each carry a full load of 3,310 kg, as shown in Table 3. However, ISS Program officials noted because the Dragon’s pressurized cargo module is volume-limited and has yet to transport more than 2,024 kg on a mission, this may not be attainable. Table 3: SpaceX CRS-1 Projected Future Mission Upmass Values SPX-9 SPX-10 SPX-11 SPX-12 SPX-13 SPX-14 SPX-15 Pressurized cargo upmass (kg) 2,023 2,029 1,737 2,349 2,333 2,760 2,410 Unpressurized cargo upmass (kg) 550 977 1,573 961 977 550 900 Total upmass (kg) 2,573 3,006 3,310 3,310 3,310 3,310 3,310 Source: NASA OIG presentation of SpaceX data. SpaceX returned its Falcon 9 to flight in December 2015 for the first time since the SPX-7 failure with the successful launch of a commercial satellite payload for ORBCOMM, Inc. Thereafter, the company launched the Jason-3 mission for NASA in January 2016 and launched another commercial satellite in March 2016.27 On April 8, 2016, SpaceX resumed cargo deliveries to the ISS with SPX-8, a resupply mission that transported about 3,200 kg of critical supplies and payloads for the Station, including materials to support science and research investigations. Dragon’s unpressurized trunk carried the Bigelow Expandable Activity Module (approximately 1,500 kg), which will attach to the Station and demonstrate expandable in-space habitat technology. Authority and Process for Investigating CRS-1 Launch Failures The authority and process for conducting an investigation after a CRS-1 launch failure is informed by FAA licensing requirements, CRS-1 and LSP contract requirements, and NASA policy. As part of its launch license application, SpaceX was required to submit an accident investigation plan to the FAA for approval prior to the SPX-7 launch. According to FAA regulations and SpaceX’s plan, the SPX-7 failure was categorized as a “launch mishap” because it involved a loss of more than $25,000 for the payload and vehicle.28 In accordance with the company’s accident investigation plan, SpaceX formed an Accident Investigation Team to determine the cause of the failure and identify any corrective actions needed to 26 The original January 2009 task order was issued for missions SPX-1 through SPX-12. Subsequently, the CRS-1 contract was modified in late 2014 (before the SPX-7 failure) to add three missions (SPX-13 through SPX-15). In December 2015, flights SPX-16 through SPX-20 were ordered at a discounted price to help compensate for the SPX-7 failure. 27 Jason-3 is the fourth mission in a U.S.-European series of satellite missions that measure the height of the ocean surface. This mission used an older version of the Falcon 9 rocket and not the improved thrust Falcon 9 system used on the ORBCOMM, Inc launch. 28 14 C.F.R. § 401.5. There are two other forms of FAA classification, a “launch accident” and a “launch incident.” A launch accident occurs if there is a fatality or serious injury, more than $25,000 of property damage to a third-party, or unplanned impact of the vehicle or payload outside designated impact limit lines. Per the FAA’s Memorandum of Understanding with the National Transportation Safety Board (NTSB) and USAF, NTSB will investigate launch accidents occurring during commercial space launches. A launch incident is an unplanned event that is not a launch accident but is still a malfunction or failure of a critical system. NASA Office of Inspector General IG-16-025 9 prevent a recurrence.29 SpaceX’s investigation board was chaired by a SpaceX official, included 10 additional company employees and 1 FAA employee.30 In addition, officials from the FAA, NASA, National Transportation Safety Board (NTSB), and USAF served as nonvoting observers.31 In order for the Falcon 9 to return to flight, the FAA had to approve the SpaceX investigation team’s findings and any corrective action plans. As noted previously, the team submitted its final report to the FAA in November 2015 with the finding that a strut assembly failure in the rocket’s second stage was the most probable cause of the launch failure. Following its review of the report, the FAA issued SpaceX a new launch license 3 days before the December ORBCOMM launch. Separate from the FAA requirements, the CRS-1 contract required SpaceX to submit an accident investigation plan to NASA. Pursuant to the plan, if a failure occurs during launch but before reaching the ISS, SpaceX is responsible for the investigation, although NASA has discretion to conduct its own, independent investigation as well. After the SPX-7 failure, NASA initiated an investigation through LSP’s contract authority rather than based on its CRS-1 contract authority as it had in the Orb-3 mishap. NASA was able to call on LSP because LSP had an existing contract with SpaceX to fly the Jason-3 payload on a Falcon 9. Before using a particular launch vehicle for a NASA mission, LSP certifies the vehicle for flight through insight and approval processes.32 The LSP investigation confirmed SpaceX’s implementation of corrective actions before approving the January 2016 Jason-3 launch. Finally, NASA has a policy for investigating launch failures that requires the formation of a Mishap Investigation Board to identify the root cause, improve safety, and prevent recurrence.33 The policy requires the majority of Board members be independent from the investigated activity, the chairperson be independent of the underlying program, no member be involved in the direct management of the activity under investigation or have a vested interest in the outcome of the investigation, and no contractor be a member of the investigation. While the CRS-1 contract cites the policy as a reference document, ISS Program officials told us NASA’s official mishap investigation policy is not applicable to CRS launches. Instead NASA adopted a tailored approach for commercial delivery services pursuant to which investigations are conducted by contractor-led boards, while the Agency retains authority to conduct its own separate investigation. Although the LSP investigation of the SPX-7 failure and the NASA investigation of the Orb-3 failure were not conducted by official Agency Mishap Investigation Boards, NASA officials have characterized the reviews as similar in purpose and process. 29 This accident investigation plan is required by the FAA. 30 Only the 11 SpaceX board members signed the final accident investigation report. 31 In comparison, the seven-member contractor board that investigated the Orb-3 failure included four Orbital employees, two NASA employees, and one third-party expert. 32 As of July 2015, LSP had certified the Falcon 9, Orbital’s Pegasus XL and Minotaur-C, and United Launch Alliance’s Atlas V and Delta II. Orbital’s Antares launch vehicle is not certified by LSP. Accordingly, following the Orb-3 failure NASA did not call on LSP to investigate but rather utilized its CRS-1 contractual authority to form an independent review team, which issued its report in October 2015. NASA, “NASA Independent Review Team Orb-3 Accident Investigation Report: Executive Summary,” October 9, 2015. 33 NASA Procedural Requirements (NPR) 8621.1B, “NASA Procedural Requirements for Mishap and Close Call Reporting, Investigating, and Recordkeeping,” July 15, 2013. NASA defines direct cause as the events that occurred, including any conditions that existed immediately before the undesired outcome; directly resulted in its occurrence; and, if eliminated or modified, would have prevented the undesired outcome. In contrast, the root cause is one of typically multiple factors that contributed to or created the direct cause and subsequent undesired outcome and, if eliminated or modified, would have prevented the undesired outcome. Root cause also includes a review of programmatic and organizational contributing causes. NASA Office of Inspector General IG-16-025 10 FAILURE CONTRIBUTED TO CARGO SHORTFALL AND MORE FLIGHTS IN FISCAL YEARS 2016 AND 2017 AND RESULTED IN LOSS OF DOCKING ADAPTER NASA lost 690 kg of crew supplies in SPX-7, including food, oxygen, and other consumables, and had to rearrange its manifests for subsequent missions to replenish consumable reserves. Furthermore, the mishaps and resulting delays while SpaceX and Orbital investigated and received permission to resume flights mean there could be as many as three more cargo missions on average in fiscal years (FY) 2016 and 2017 than were flown in previous years.34 Finally, loss of the Docking Adapter means the ISS will lack a redundant port for the Commercial Crew demonstration missions scheduled to begin in May 2017. Upmass Shortfall and Additional Cargo Missions in FYs 2016 and 2017 To replenish supplies lost as a result of the SPX-7 failure and ensure astronaut safety, the ISS Program adjusted the manifests for subsequent missions to restore on-orbit consumable reserves. Furthermore, the increased number of missions in 2016 and 2017 will require astronauts to devote more time to unloading and loading vehicles and therefore likely allow them less time to conduct research. Shortfall in Pressurized Upmass Impacts ISS Supplies Due to the loss of SPX-7 and the shift of SPX-8 to 2016, approximately 3.48 metric tons of pressurized upmass scheduled for delivery in FY 2015 did not arrive on the Station. NASA was able to absorb this loss because increased packing efficiencies and high cargo densities enabled transport of an additional 746 kg of upmass on two FY 2015 SpaceX cargo missions (SPX-5 and SPX-6) and one Japan Aerospace Exploration Agency cargo mission (HTV-5). See Table 4 for details on SpaceX’s upmass performance per mission. In addition, the Russian space agency – Roscosmos – carried an additional 100 kg of pressurized upmass for NASA over six different FY 2015 flights. These measures reduced the total upmass shortfall for FY 2015 by 24 percent, from 3.48 metric tons to 2.63 metric tons.35 Furthermore, in March 2016, Orbital’s Orb-6 mission delivered 3,602 kg to the ISS – the most of any CRS mission to date. 34 This average is only for cargo missions to the U.S. segment of the ISS and does not include cargo deliveries to the Russian segment. 35 In contrast to the upmass shortfall for 2015, NASA is projecting a 2.2 metric ton excess in upmass capability in 2016. ISS Program managers told us this figure could increase due to packing efficiencies and the additional upmass capacity of the two Orbital missions that launched on Atlas V rockets, which are capable of lifting heavier loads than Orbital’s Antares rocket. NASA Office of Inspector General IG-16-025 11 Table 4: SpaceX CRS-1 Mission Upmass Values SPX-1 SPX-2 SPX-3 SPX-4 SPX-5 SPX-6 SPX-7 SPX-8 Pressurized cargo upmass (kg) 450 644 1,629 1,729 1,900 2,024 1,952 1,707 Unpressurized cargo upmass (kg) 0 221 487 609 494 0 526 1,552 Total upmass (kg) 450 865 2,116 2,338 2,394 2,024 2,478 3,259 Source: NASA OIG presentation of ISS Program data. Among the cargo that was not delivered in 2015 were consumables (such as food, oxygen, water, and nitrogen), crew supplies, utilization equipment, and vehicle hardware. The ISS Program strives to maintain a significant store of consumables on-orbit in reserve for redundancy, including a 6-month supply of food (see Table 5). Table 5: ISS Consumable Reserves Reserve Reserve Reserve Assumptions Requirement Supports six crew members consuming 0.8 rations per day per crew member for Food 216 rations 45 days. Russian and U.S. rations are shared when either side reaches zero. Supports U.S. Orbital Segment reserve (6 month’s usage, nominal and contingency Oxygen 290 kg EVAs) and Russian reserve (45 days with three crew members). Supports U.S. Orbital Segment reserve (55 days no regeneration capability) and Water 1034.5 liters Russian reserve (45 days with three crew members). Supports repressurization of largest module – currently the Japanese Experiment Nitrogen 132 kg Module – and 6 months of nominal use. Source: NASA OIG presentation of ISS Program data. The SPX-7 mission carried 690 kg of crew supplies intended to replenish these reserves. Although the loss of these supplies did not pose immediate danger for the ISS astronauts, NASA was forced to make changes to its cargo manifests for subsequent missions. Increase in FYs 2016 and 2017 Missions May Reduce Time Available for Research The SpaceX and Orbital mission failures have led to a compressed launch schedule in FYs 2016 and 2017, with 11 CRS-1 missions, 7 Russian cargo missions, and 1 Japanese cargo mission now scheduled to arrive at the Station.36 In addition, SpaceX’s first commercial crew demonstration mission is scheduled for May 2017. NASA devotes a minimum of 35 hours of crew time a week to research and other utilization activities, but in mid-2014 the astronauts were spending as much as 44 hours a week on these activities. Although program officials stated that the number of research hours will not fall below the 35-hour minimum, as astronauts take time to receive, unpack, and repack all of these vehicles the total time devoted to research may decrease from 2014 levels. See Figure 2 for the cargo mission schedule as of June 2016. 36 Agency officials told us this flight schedule is not set in stone and may change. NASA Office of Inspector General IG-16-025 12 Figure 2: ISS Cargo Mission Schedule for FYs 2015 through 2019 Source: NASA OIG presentation of ISS Program data. a Red shading indicates a failed mission. b The Russian Soyuz capsule and Progress cargo vehicle only fly small amounts of U.S. cargo. The ISS crew is also scheduled to conduct five important EVAs in FYs 2016 and 2017, including installation of new lithium-ion batteries into the Station’s power supply and installation of the second Adapter. This does not include any contingency EVAs that may become necessary. The ISS Program typically plans 6.5 hours to complete an EVA, although complex EVAs can take longer. For example, an EVA to relocate a pressurized mating adapter and install a fixture on the U.S. Destiny laboratory in 2001 lasted nearly 9 hours. If there are additional contingency EVAs or complications with the planned EVAs, research time may be further reduced. Consequences of Loss of International Docking Adapter The most significant cargo item lost during the SPX-7 mission was the first of two Adapters necessary to prepare the Station for the arrival of commercial crew missions. Although NASA had planned to have two of the Adapters installed on the Station before the first commercial crew demonstration mission scheduled for May 2017, now it is likely only one Adapter will be installed by that date. At the time of our report, NASA projected the second Adapter would fly on SPX-9, scheduled for launch in July 2016, and the replacement for the lost Adapter – which is currently being assembled from spare parts – on SPX- 14 in February 2018. However, the replacement NASA Office of Inspector General IG-16-025 13 could be delayed if the necessary parts are not available in time. Although the ISS Program has spares of each of the parts, several key items with the longest lead times to manufacture, including the metal shielding that wraps around the Adapter, need to be fabricated. Moreover, even if NASA is able to meet its planned schedule, the Station likely will have only one Adapter when the commercial crew demonstration missions are scheduled to arrive in May, August, and December 2017, and February 2018.37 (See Figure 3.) Figure 3: Launch Schedule for Adapters and Commercial Crew Demonstration Missions Source: NASA OIG analysis of ISS and Commercial Crew Program information. ISS Program officials acknowledged that redundancy is important for docking systems and that having only one Adapter means that a commercial crew vehicle will not be able to dock with the ISS if technical issues arise with the single available docking port. ISS Program officials stated that they plan to have the replacement Adapter installed before regular commercial crew rotations begin. 37 Crew demonstration mission dates are based on the ISS Flight Schedule as of June 2016 and are subject to change. We are conducting a separate audit of the Commercial Crew Program. NASA Office of Inspector General IG-16-025 14 NASA BENEFITED FROM DISCOUNTED MISSION PRICING AND NEGOTIATED VALUABLE CONSIDERATION, BUT HAS NOT FULLY UTILIZED SPACEX CARGO CAPACITY NASA is effectively managing the SpaceX CRS-1 contract to reduce cost and financial risk. The Agency has taken advantage of multiple mission pricing discounts and negotiated and incorporated into the CRS-1 contract equitable adjustment modifications of significant value to the Agency. In addition, following the SPX-7 failure, NASA negotiated from SpaceX significant consideration in the form of Adapter hardware, integration services, manifest flexibility, and discounted mission prices for missions SPX-16 through SPX-20. However, for the first seven CRS-1 missions NASA has not fully utilized SpaceX’s unpressurized cargo capability and could obtain even better value by filling the Dragon 1’s unpressurized trunk to capacity for the remaining CRS-1 flights.38 NASA Negotiated Favorable Pricing for Initial Flights and Contract Extensions As of June 2016, SpaceX has successfully completed 7 of 20 planned deliveries to the ISS under the CRS-1 contract.39 NASA has paid SpaceX a total of $1.7 billion for resupply services, as of March 31, 2016. All of these flights were priced in accordance with the mission pricing tables in the CRS-1 contract and included both upmass and downmass capabilities. By using the mission pricing tables, NASA received discounted pricing when ordering multiple flights for a single year. For example, at the time of the original CRS-1 contract award, had NASA purchased two flights to be flown in 2015 the price for each flight would have been less than if the Agency paid for a single flight. If the Agency purchased three, the price was further reduced. Additionally, NASA protected its financial interests in the event of launch delays caused by SpaceX by locking in the rate for the year a flight was scheduled to launch rather than paying the higher rate applicable to the actual launch year. For example, although only one of three flights planned for calendar year 2013 actually launched that year, NASA paid the same price for the delayed flights. By holding SpaceX to the original prices for SPX-3 through SPX-7, NASA saved more than $65 million.40 Table 6 reflects delays in SpaceX’s flight schedule for missions SPX-1 through SPX-7 from the launch plan negotiated in 2009 to the actual flight schedule. 38 The SPX-8 mission carried the Bigelow Expandable Activity Module in the trunk and was volumetrically full. 39 SpaceX also flew demonstration missions in December 2010 and May 2012. 40 NASA also locked in rates for the year the flight was scheduled in the Orbital CRS-1 contract. However, as discussed in our September 2015 report on the Orb-3 failure (IG-15-023), because of the way the contract with Orbital was structured NASA could have obtained lower prices by invoking a clause that provided for multiple flight discounts. NASA Office of Inspector General IG-16-025 15 Table 6: Planned vs. Actual Launch Dates, by Calendar Year 2012 2013 2014 2015 Planned launches 2 3 3 2 Actual launches 1 1 2 3 Source: NASA OIG analysis of ISS Program and SpaceX information. In September 2014, NASA modified the SpaceX CRS-1 contract to extend the period of performance through December 2016, added mission pricing for calendar years 2017 and 2018, and ordered SPX-13 and SPX-14. NASA ordered SPX-15 in December 2014. In 2015, the contract was modified once again to extend the period of performance through December 2018. In determining reasonable mission prices for calendar years 2017 and 2018, NASA conducted a detailed price analysis and appropriately documented the rationale for its negotiation position. The analysis included evaluating proposed inflation and escalation rates against historical inflation rates and industry standards, comparing SpaceX’s proposed prices with those of other launch service providers, and obtaining an independent Government cost estimate from the ISS Assessments, Cost Estimating, and Schedules Office. NASA proposed and SpaceX agreed to a minimal price increase from calendar year 2016 prices for (1) the labor costs associated with special task assignments and studies, and (2) the evaluation costs of NASA cargo manifest and payload changes. Finally, NASA negotiated inclusion of additional capabilities not part of the original mission price. NASA Effectively Negotiated Contract Modifications and Received Good Value for Adjustments The SpaceX CRS-1 contract provides that in the event of a launch delay of more than 30 days, regardless of cause, the NASA contracting officer shall request information from the company about the effect of the delay on price, schedule, and other contract terms relating to the affected mission. The contract further states that this exchange between NASA and SpaceX “may result in an equitable adjustment to the price of all contract line item numbers in the task order (if any), change in the delivery schedule, and change in the period of performance.” If NASA and SpaceX fail to agree to an adjustment, the contracting officer may unilaterally adjust the task order. In the aftermath of the SPX-7 failure, NASA and SpaceX negotiated an equitable adjustment to compensate NASA for launch delays resulting from the failure. Most notably, SpaceX agreed to provide at no additional cost significant enhancements to the Agency’s science and operational capabilities. In addition to the adjustments that followed the SPX-7 failure, we found NASA has consistently negotiated equitable adjustments throughout the life of its CRS-1 contract with SpaceX. For each equitable adjustment, NASA officials performed a technical analysis to determine the value of an adjustment to the ISS Program. NASA officials indicated, and we confirmed, that all equitable adjustments provided NASA with either additional capabilities at no increase in cost or intangible benefits of value to the ISS Program and the research community, or both. In each case, NASA clearly explained how the consideration represented value to the ISS Program and the manner in which additions or enhancements could be quantified. In addition, NASA officials indicated that nonmonetary benefits, while not quantifiable, are just as important to the Agency and the science and research community, or in some cases, more important than dollars saved. For example, increasing the powered NASA Office of Inspector General IG-16-025 16 capability of the Dragon 1 is significant because the majority of science experiments – in particular the transportation of live animals – requires power throughout the launch, flight, and return phases of the mission. By increasing powered capability, SpaceX tripled the number of powered payloads that could be accommodated, which provides a significant enhancement to ISS science capability. A by-product of this redesign is the ability to reallocate spacecraft power between internal and external payloads on a flight by flight basis, adding more flexibility to accommodate various types of payloads. NASA Did Not Fully Utilize SpaceX’s Upmass Capability Despite the value NASA has realized from the SpaceX CRS-1 contract, the Agency could have benefited even more had it fully utilized the Dragon 1’s unpressurized trunk during the spacecraft’s cargo missions. For example, on SPX-3 through SPX-7, NASA flew only 423 kg on average of unpressurized cargo, even though the trunk is capable of carrying significantly more cargo per mission.41 As shown in Figure 4, SPX-1 and SPX-6 launched with empty unpressurized trunks due to performance issues (SPX-1) or payload issues (SPX-6). Furthermore, according to the CRS-1 contract, the Dragon 1’s unpressurized trunk is capable of transporting up to three Flight Releasable Attachment Mechanisms (FRAM) – the standard mounting mechanism for transporting unpressurized upmass on the capsule.42 However, as depicted in Figure 4, although the unpressurized trunk had the capacity to fly one or more additional FRAMs on five of the seven SpaceX missions, NASA did not provide the additional cargo. Only SPX-2 and SPX-7, both of which featured a custom-built trunk, were volumetrically full. NASA told us there are numerous reasons for not flying full unpressurized payloads including manifest priority, payload availability, and mission risk. Figure 4: Unpressurized Cargo for SPX-1 through SPX-7 Source: NASA OIG analysis of ISS Program information. Note: SPX-1 and SPX-2 carried limited upmass due to performance issues and SpaceX provided consideration for the reduced upmass on these flights. SPX-3 was the first robotic extraction of a FRAM from the Dragon 1’s trunk, and SPX-4 and SPX-6 experienced issues with payload readiness. SPX-5 was the first SpaceX mission to transport an Exposed Facility payload. As a result, the ISS Program did not know that it was able to fly an additional FRAM. 41 The Dragon 1 is capable of transporting up to 3,310 kg of upmass, which can include pressurized or unpressurized cargo or a combination of the two. Priority is typically given to pressurized cargo, limiting the weight that can be transported in the unpressurized trunk. 42 Alternatively, the Dragon 1’s trunk can carry one Exposed Facility and one FRAM. An Exposed Facility is an external platform that can hold up to 10 experiments for use on the Japanese Experiment Module. NASA Office of Inspector General IG-16-025 17 Nevertheless, by underutilizing the Dragon 1’s unpressurized trunk the ISS Program did not maximize the value of these specific SpaceX missions. Under the terms of the CRS-1 contract, NASA is responsible for manifesting cargo, and therefore, as long as SpaceX supplies a spacecraft that meets contractual requirements, the Agency is not due any consideration if the vehicle flies with less than a full payload. In other words, NASA must pay full value for the mission whether the Agency produces 0 kg of unpressurized cargo, as it did for SPX-6, or 1,552 kg of unpressurized cargo, as it did for SPX-8. The ISS Program acknowledged it struggled to utilize the Dragon 1’s trunk on the early CRS-1 missions, noting that after the Space Shuttle retired a gap in procurement and planning for this type of payload existed while the commercial partners were developing transportation capabilities. As a result, appropriate payloads were not ready at the time the SpaceX missions flew. ISS Program officials are seeking to increase the amount and weight of unpressurized payloads being built, and as of June 2016, Agency manifests show full trunks on all future mission manifests, flying three-FRAM and custom payloads. As of the time of our audit, SpaceX and NASA projected unpressurized payloads for missions SPX-9 through SPX-15 will average 927 kg per mission, with one unpressurized payload of approximately 1,500 kg. In our judgment, NASA’s efforts to ensure it maximizes the Dragon’s payload capacity are reasonable and, if executed as planned, will improve the value NASA receives from the CRS-1 contract. NASA Office of Inspector General IG-16-025 18 NASA COULD IMPROVE RISK MANAGEMENT FOR COMMERCIAL CARGO RESUPPLY MISSIONS One of the goals of the CRS-1 contract was to achieve reliable, cost effective access to low Earth orbit while creating a market environment in which commercial space transportation services are available to Government and private sector customers. In 2008, in line with this goal and before awarding the CRS-1 contracts, NASA adopted a tailored risk management approach for cargo missions that deviates from existing Agency risk classification processes. In practice, NASA has informally treated all CRS payloads as the lowest level risk classification irrespective of value and relied primarily on SpaceX and Orbital to evaluate and mitigate launch risks. In our judgment, NASA’s practice does not provide sufficient information to Agency management regarding the risks associated with a particular launch. The Independent Review Team that examined Orbital’s October 2014 launch failure raised similar concerns, and the ISS Program is considering how to address the Team’s recommendations to increase insight, improve communication, and enhance risk management. ISS Program Deviated from Existing Agency Risk Processes NASA generally uses the following processes to categorize payload risk and certify launch vehicles: Risk Classification for Payloads. This process categorizes payload risk as class A (high) through class D (low) and provides a structured approach for defining a hierarchy of risk combinations for payloads by considering such factors as availability of alternative research or reflight opportunities, success criteria, and magnitude of investment.43 See Appendix B for more information on the four classifications. Launch Services Risk Mitigation. This certification process categorizes launch vehicle risk as 1 (high), 2 (medium), or 3 (low) in conjunction with the payload classification and sets parameters for using a particular launch vehicle, such as flight experience and testing, verification, and risk management activities.44 Details concerning these three categories and selected criteria used in evaluating a launch vehicle are included in Appendix B. For cargo missions, the ISS Program adopted a tailored approach that deviates from these processes.45 Specifically, the Program has informally treated CRS-1 cargo as class D payloads, meaning, among other things, that the cargo is low priority and low cost and has few to no launch constraints and significant alternative or reflight opportunities. Because under NASA risk management policies “high-risk” launch 43 NPR 8705.4, “Risk Classification for NASA Payloads,” October 2, 2014. 44 NASA Policy Directive (NPD) 8610.7D, “Launch Services Risk Mitigation Policy for NASA-Owned and/or NASA-Sponsored Payloads/Missions,” August 27, 2012. 45 NASA convened a Special Flight Planning Board to approve the classification of CRS-1 flights as “on-orbit delivery services” as opposed to a launch services in accordance with NPD 8610.7D. This decision held the contractor responsible for mission success and significantly reduced NASA’s technical assessment of launch readiness and launch risk through the Agency’s existing processes. NASA Office of Inspector General IG-16-025 19 vehicles may carry only class D payloads, classifying CRS-1 cargo in this way provides the Program more flexibility to use high-risk launch vehicles. However, this approach results in nebulous risk classifications not defined in NASA policy. For example, NASA labeled SpaceX’s return to flight mission – SPX-8 – as posing an “increased risk.” Similarly, NASA described the Orb-3 mission as posing an “elevated but acceptable risk.” In our view, using a more formal risk categorization approach for CRS-1 missions would better inform Agency management about the risk level of particular missions and allow for consideration of possible ways to mitigate associated risks such as requesting additional testing or, as suggested to us by a former program engineer in relation to the Orb-3 flight, that the company adjust the throttle to exert less force on the engines. NASA Primarily Relies on SpaceX and Orbital to Evaluate and Mitigate Launch Risk The ISS Program heavily relies on SpaceX and Orbital to assess and mitigate risk for launches, with the Agency endeavoring to protect its cargo through an “insight clause” in the contract. For the CRS-1 contracts, NASA insight is defined as gaining an understanding necessary to knowledgeably assess the risk of contractor actions or lack thereof through observation of manufacturing or testing, review of documentation, and attendance at meetings and reviews. As part of the insight process, NASA conducts a technical assessment of the readiness and risk posture for each launch.46 However, ISS Program officials told us there is no integrated presentation or package that documents all risk areas for a given launch.47 Instead, separate presentations are used to determine the “acceptable” risk posture – a term that evolves frequently.48 An acceptable risk may be based on such factors as the level of reserves and supplies aboard the ISS, the need to deliver or return research, or the timing of upcoming scheduled flights. For example, the successful Orb-4 and Orb-6 missions provided key supplies and research and restored consumable reserves on the Station thereby reducing the pressure to fly SPX-8 and giving NASA flexibility to accept a move in the launch date of that mission from January to April 2016, which in turn provided SpaceX more time to implement corrective actions before returning to flight. Although the flexibility in determining and altering the nature of an acceptable risk posture has some benefits, it may also introduce confusion into the process. For example, senior NASA officials have stated that high levels of risk for cargo missions are tolerable, noting the expected risk of mission failure for a typical CRS-1 launch is one in six. However, as stated in the Orb-3 Independent Review Team’s report, NASA engineering personnel expressed significant concerns about the Orb-3 launch vehicle’s engines and the recent failures Orbital had experienced on test stands, characterizing the likelihood of mission failure for Orb-3 as “50/50.” In contrast, the ISS Program’s risk matrix reflected the risk of Orb-3 engine issues as “low” and assigned a subjective risk of “elevated but acceptable.” Although according to some ISS Program officials NASA management is generally willing to accept heightened risk for cargo missions, it is unclear whether senior NASA management clearly understood the increased likelihood of 46 The primary tool for maintaining insight and determining the risk posture for a commercial resupply launch is the Launch Vehicle Assessment, which helps NASA understand the level of risk associated with a CRS-1 launch. 47 NASA only conducts formal reviews for risks related to the spacecraft’s approach to the ISS for berthing and docking. 48 An acceptable risk posture for a specific launch can change based on numerous logistical constraints, on-orbit supply needs, and the vehicle's capabilities, constraints, and risks. NASA Office of Inspector General IG-16-025 20 failure for the Orb-3 mission. Even so, the disparity between 50/50 and one in six for the same mission raises questions about the adequacy of communication between the engineers and top program management. In our judgment, the absence of a multi-disciplined approach to launch readiness, such as identifying and understanding all launch vehicle and payload issues and assigning a more objective launch rating to the mission to aid in communication of the risk, hampers successful risk mitigation efforts. We believe ISS Program officials could benefit from exploring USAF’s experience with the Evolved Expendable Launch Vehicle Program.49 USAF officials told us that after a series of launch failures in the late 1990s, they applied a more disciplined approach to launch mission assurance. Adjustments to the depth and priority of the required insight in specific areas happened only after the contractors had a proven track record of success.50 Furthermore, USAF continues to assess its missions according to the severity and probability of occurrence and inputs those risks into a matrix to formally communicate the risk of failure to senior leadership. (See Appendix C for more details.) ISS Program officials and officials in NASA’s Office of Safety and Mission Assurance agreed that a more regimented approach to communicating risk would benefit the ISS Program. Orb-3 Review Team Expressed Concerns about Risk Management Although careful to point out that the results of its review were specific to Orbital, the report by the Orb-3 Independent Review Team mirrored our concerns regarding communication and management of risk and made several programmatic recommendations intended to ensure the success of future cargo missions. Those recommendations included 1. formally defining and communicating a baseline level of acceptable risk and a particular launch vehicle risk to ensure personnel throughout the program are assessing issues to a consistent risk level; 2. establishing a standing working group for parties with launch vehicle responsibilities to openly discuss and coordinate launch vehicle issues and status of risk assessment activities; 3. reassigning responsibility for launch vehicle assessment to a senior engineer at Marshall Space Flight Center; and 49 The Evolved Expendable Launch Vehicle Program was initiated by USAF in 1994 to reduce costs, improve reliability, and create a more “commercial-like” procurement process. The Program consists of the Atlas V (formerly provided by the Lockheed Martin Corporation) and the Delta IV (formerly provided by Boeing) families of launch vehicles and is similar to CRS-1 services in that the Government is purchasing a launch service commercially. 50 The launch vehicle families developed for the Evolved Expendable Launch Vehicle program have successfully launched 93 Atlas V and Delta IV vehicles from 2002 to March 31, 2016. Of those launches, 59 were National Security Space missions that were certified for flight worthiness by USAF. USAF officials stated the Evolved Expendable Launch Vehicle Program was initiated in 1994 under the assumption that mission assurance would be achieved through a high commercial launch rate. However, in the late 1990s there were several commercial and USAF launch failures and the commercial launch market collapsed, which caused USAF to transition from the original commercial-like approach to the increased role of a Government launch readiness verification and certification process. The creation of the launch verification matrix process – a process in which launch readiness verification activities are planned, executed, and recorded – is an example of this increased role. Currently, the USAF has a comprehensive insight role into their contractors’ activities and the launch verification matrix is reviewed for efficient and effective mission assurance with each launch provider. NASA Office of Inspector General IG-16-025 21 4. applying risk reduction lessons learned from CRS-1 and related development activities.51 As of December 2015, the ISS Program had decided not to implement recommendations 3 and 4, stating that the lead for launch vehicle assessments needs to be within the Program Office at Johnson Space Center and that implementation of the fourth recommendation as written would be too expensive. Program officials indicated they intend to move forward with cargo missions while considering how best to implement the remaining recommendations. 51 For instances in which NASA intends to share development costs and risks with commercial industry similar to the CRS-1 contract approach, the Independent Review Team recommended the Agency (1) allow system development and demonstration efforts to be complete before establishing fixed-price contracts; (2) perform greater due diligence for major system components, including a review of contractual relationships and integration plans between the service providers and their contractors; and (3) include contract provisions to require integrated partnerships between service providers and engine providers. We expressed concerns related to this recommendation in our September 2015 report on the Orb-3 failure (IG-15-023). NASA Office of Inspector General IG-16-025 22 STANDARDIZATION AND INDEPENDENCE REQUIREMENTS COULD IMPROVE CRS-1 MISHAP INVESTIGATIONS While NASA had multiple plans to investigate the SPX-7 and Orb-3 failures, the Agency does not have an official, coordinated, and consistent mishap policy for CRS-1 launches. The lack of standardization could affect the ability of NASA and its contractors to determine the root cause of a launch failure and implement corrective actions. Additionally, as noted in our report on the Orbital failure, we have concerns about the independence of contractor-led mishap investigations. NASA Could Improve Investigation Policies and Coordination for CRS-1 Launches NASA’s official policy for mishap investigations does not directly address the process for failures of FAA-licensed commercial space launches.52 As a result, when a CRS-1 mission fails NASA determines on a case-by-case basis whether to form an ad hoc investigation through its various discretionary authorities (as shown in Appendix D, Figure 7 and Table 12). This determination is based on FAA licensing requirements, CRS-1 and LSP contract requirements, and the ISS Contingency Action Plan. As such, for the SPX-7 launch failure there were up to seven possible investigation authorities depending on when the failure occurred and the extent of damage to coordinate and prioritize.53 Due to a lack of standardization or NASA policy, the contractor and NASA investigations into the SPX-7 and Orb-3 failures had different scopes and produced varying findings and corrective actions. The findings of the contractor-led investigation boards were generally limited to determining the “technical cause” of the failures and implementing corrective actions to replace failed parts or systems.54 NASA’s investigations had broader objectives, but varied in scope and purpose. For example, LSP not only evaluated the technical causes of the SPX-7 failure but also made findings related to the selection, use, and lack of testing of the failed strut assembly as well as a general finding recommending additional measures when using commercial grade parts on launch vehicles. 52 For failures of NASA-owned or -operated launches, Agency policy provides that NASA form a Mishap Investigation Board to determine the root cause and recommend corrective actions. However, this policy only applies to CRS-1 launches once the spacecraft reaches the proximity of the ISS. 53 We also found that four different accident investigation plans existed among NASA, the FAA, and SpaceX (see Appendix D, Figure 6). 54 Technical cause is the condition that directly resulted in the failure and is usually limited to determining what physical part or system literally caused the failure. Orbital’s investigation for Orb-3 recommended replacing the whole engine system and SpaceX’s investigation for SPX-7 recommended replacing the failed strut part. NASA Office of Inspector General IG-16-025 23 While the Agency’s SPX-7 and Orb-3 investigations had elements of a traditional NASA Mishap Investigation, they were not as comprehensive as the process described in NASA policy. For example, the Orb-3 Independent Review Team made programmatic recommendations to the ISS Program, while the LSP SPX-7 Investigation did not. In addition, neither of the investigations was directed to determine all elements of a full “root cause” determination – defined by NASA as determination of the cause of the failure, including technical, organizational, and programmatic issues by reviewing the actions of the contractor and all related parties.55 Accordingly, the Orb-3 Independent Review Team used root cause analysis to develop a fault tree that included findings and recommendations related to programmatic and organizational issues, while the LSP SPX-7 investigation team stated they did not conduct root cause analysis but rather focused on the technical aspects of the failure. Had NASA undertaken an official Agency Mishap Investigation for the failures, Agency policy would have required a root cause analysis with comprehensive corrective actions directed at the contractor and the ISS Program to prevent the specific technical cause from reoccurring and to address any programmatic weaknesses that contributed to the failure. Table 7 compares the four investigations to the NASA Mishap Investigation standards. Table 7: Comparison of Investigations’ Scope and Findings SpaceX Orbital Orb-3 NASA Scope of Findings and (contractor- LSP SPX-7 (contractor- Independent Mishap Recommendations led) Investigation led) Review Investigation Investigation Investigation Team Standards Technical causea (limited to physical part/system that failed, resulting in the failure) Corrective actions to replace the part or system resulting in the technical cause Findings on contractors’ programmatic issues –b Findings to improve NASA’s CRS contract management Source: NASA OIG analysis of CRS investigation reports and NASA policy. a NASA policy also refers to technical cause as proximate or direct cause. NASA Procedural Requirements 8621.1B. b While SpaceX did not report on organizational or programmatic causes to the accident, its corrective actions spanned programmatic and organizational steps to improve the company’s posture and alleviate the programmatic structures that allowed the failure to occur. 55 Root cause is an event or condition that is an organizational factor that existed before the technical cause and directly resulted in its occurrence (thus indirectly it caused or contributed to the proximate cause and subsequent undesired outcome) and, if eliminated or modified, would have prevented the technical cause from occurring and the undesired outcome. Typically, multiple root causes contribute to an undesired outcome. Root cause analysis is a structured evaluation method that identifies the root causes for an undesired outcome and the actions adequate to prevent recurrence. Root cause analysis should continue until organizational factors have been identified or until data are exhausted. NASA Office of Inspector General IG-16-025 24 While a complete NASA Mishap investigation is not required for launch failures under the terms of the CRS-1 contract, in our judgment the absence of more formal guidance for CRS investigations increases the risk that contractor corrective actions may not fully address broader contributing causes. In addition, NASA lacks a memorandum of understanding with the FAA to coordinate and delegate accident investigation authority during CRS launches involving the FAA, NASA, NTSB, USAF, and contractor.56 After the Orb-3 failure, there was confusion among FAA, NASA, and Orbital on how to immediately respond and impound evidence. While these issues were resolved relatively quickly, NASA officials identified the need for a more formalized understanding between all the affected parties involved in an FAA-licensed commercial space launch failure. Moreover, FAA officials stated there is the potential for the FAA to relicense a company’s launch vehicle before reviewing NASA’s independent investigation of the failure. For example, although FAA officials had access to LSP meetings during the SPX-7 investigation, the FAA did not receive LSP’s final report with findings and recommendations for corrective actions before SpaceX obtained FAA approval to return to flight in December 2015. While the FAA was not required to review LSP’s findings before issuing a license, this uncoordinated approach increased the risk the FAA approved a launch without fully understanding the LSP investigation’s findings and recommended corrective actions. Moreover, according to officials from NASA’s Office of Safety and Mission Assurance, the Agency will always conduct an independent investigation of failures of commercial launches. As such, the Office is currently in the process of developing a memorandum of understanding to better coordinate with all relevant parties and is updating the NASA mishap policy to clarify the process for independent investigations. Lack of Independence Could Inhibit Contractor-Led Investigations In our report examining NASA’s response to the Orb-3 launch failure, we found that Orbital’s Accident Investigation Board was not independent.57 Similarly, we found SpaceX’s investigation board was not independent because 11 of the 12 voting members were SpaceX employees.58 While not required in the CRS-1 contract, this lack of independence does not meet Government best practice standards for NASA, NTSB, and USAF investigations and could impact the board’s ability to identify the root cause and make corrective actions. NASA’s official policy for investigations requires all official Mishap Investigation Boards to be independent.59 NTSB and USAF have similar requirements.60 In contrast with these best practices, the 56 In 2004, the FAA, NTSB, and USAF formed a Memorandum of Understanding, which did not include NASA, before the CRS-1 contract was initiated. 57 NASA OIG, IG-15-023. 58 SpaceX’s FAA-required investigation plan requires the chairperson and board members to be impartial in their analysis. Although there was no definition of impartiality in the investigation plan, SpaceX explained that the impartiality requirement means board members are able to evaluate evidence critically and objectively to reach conclusions without being subject to financial, political, legal, or interpersonal influences. 59 NPR 8621.1B. Specifically, the board chair must be independent of the activity, the majority of board members must be independent, and contractors cannot be voting board members. According to ISS Program officials, NASA’s official mishap investigation policy is not applicable to CRS launches. 60 In the event of a launch failure under the Evolved Expendable Launch Vehicle contract, USAF has authority to conduct the accident investigation. Depending on the situation and as outlined in the contract, a contractor could conduct an investigation, but USAF has monitoring standards for the contractor-led investigation board. NASA Office of Inspector General IG-16-025 25 CRS-1 contract and FAA license requires SpaceX to conduct its own investigation but does not require company investigation boards to screen for conflicts of interest or maintain independence. FAA officials stated NASA can implement additional independence requirements for contractor-led investigations through its contracts as long as they do not conflict with FAA regulations. NASA and SpaceX officials responded that specific expertise in the failed launch vehicle is important for an accident investigation board and that this factor should be taken into consideration even though it may impact the board’s independence. We agree that engineering expertise is invaluable for determining the causes of a failure and developing corrective actions. However, other processes such as NASA’s Anomaly Engineering Board or USAF’s Engineering Analysis Group are available to obtain this expertise without compromising the independence of an investigation board.61 We acknowledge SpaceX’s investigation was transparent and the observers from FAA, ISS, LSP, NTSB, and USAF had access to the investigation’s data and analysis. However, an investigation led by the employee responsible for the SPX-7 launch and run by the contractor responsible for the failure raises questions about inherent conflicts of interest. To independently verify and review the contractor investigations, NASA created its own investigation boards for both the Orbital and SpaceX CRS-1 mission failures. While NASA, Orbital, and SpaceX, have similar incentives to safely and quickly return to flight, the structure of the contractor-led investigations may not result in a full review of all programmatic and organizational contributing factors and consequently these factors may not be fully addressed to prevent future failures. 61 NASA’s official investigation policy allows for creation of a nonvoting Anomaly Engineering Board to examine technical engineering and factual issues. USAF also creates a nonvoting Engineering Analysis Group to assist its Safety Investigation Boards. LSP officials told us they had access to all required technical expertise for their analysis of the SPX-7 failure. NASA Office of Inspector General IG-16-025 26 CONCLUSION The failure of SpaceX’s seventh CRS-1 mission has resulted in a shortfall of 2.63 metric tons of pressurized cargo intended for delivery to the Station in FY 2015. In addition, one of the Adapters necessary to ready the Station for the arrival of commercial crew vehicles was destroyed, putting at risk NASA’s plans to have two Adapters installed on the ISS by the beginning of commercial crew demonstration missions in May 2017. The second Adapter would have provided for redundancy in case the first Adapter failed. Although NASA did not fully utilize the Dragon’s unpressurized cargo transportation capability during the first seven missions, the Agency plans to increase utilization for upcoming missions. In addition, NASA has effectively managed its contract with SpaceX to reduce cost and financial risk. Specifically, NASA has taken advantage of multiple mission pricing discounts and effectively negotiated and incorporated into the contract equitable adjustment modifications of significant value to the Agency. In the aftermath of the SPX-7 failure, as well as over the life of the contract, NASA effectively negotiated equitable adjustments and received adequate consideration for SpaceX not meeting contract requirements. Due to the commercial nature of cargo resupply missions, the ISS Program is not using standard procedures for evaluating launch risk. As a result, risk mitigation procedures are not consistently employed and the subjective launch ratings the Agency uses provide insufficient information to NASA management concerning true launch risks. In addition, the current process for coordinating and conducting accident investigations for CRS-1 launches lacks standardization and independence, and could adversely affect root cause analysis. NASA Office of Inspector General IG-16-025 27 RECOMMENDATIONS, MANAGEMENT’S RESPONSE, AND OUR EVALUATION In order to maintain the efficacy of the ISS and ensure delivery of cargo in a timely and affordable manner, we recommended the Associate Administrator for Human Exploration and Operations ensure the ISS Program: 1. Incorporate the risk of limited availability of the Adapter into risk management processes. 2. Continue to refine unpressurized upmass manifesting process and consider preparing alternative unpressurized upmass payloads in the event scheduled payloads cannot be launched. 3. Quantify overall mission risk ratings and communicate the risks for upcoming launches early and in coordination with varying levels of engineering and management. 4. Review all investigation authorities and plans during commercial launches with NASA payloads to ensure they are standardized. In particular, NASA should review the contract requirements, ISS Program Office plans, the FAA Accident Investigation Plan, and contractor submitted plans to ensure each references the other and are coordinated and incorporate programmatic and organizational root cause analysis. In order to clarify the division of roles and responsibilities in the event of a mission failure, we recommended the Office of Safety and Mission Assurance, in conjunction with ISS Program officials: 5. Improve coordination with other Federal agencies involved in commercial space. For example, consider a. creating a formal Memorandum of Understanding with the FAA, NTSB, and USAF to coordinate accident investigations; b. coordinating with other Federal agencies to determine the hierarchy and roles of different investigation authorities during all phases of commercial launches with NASA payloads; and c. communicating investigation findings and corrective actions to all interested Federal agencies to allow full and informed decisions. 6. Update NPR 8621.1B to include commercial space launches with NASA payloads in official mishap policies. In particular, NASA should a. define commercial space launches with NASA payloads; b. determine the extent to which official NASA mishap policies apply in commercial space launches with NASA payloads; c. describe what types of investigations may occur and the processes to be followed in lieu of an Official Mishap Investigation Board, such as an independent investigation board created by NASA; and NASA Office of Inspector General IG-16-025 28 d. clarify the scope and purpose of each investigation, such as a NASA defined root cause compared to a technical root cause analysis, and consider the inclusion of programmatic and organizational root cause analysis. We provided a draft of this report to NASA management who concurred or partially concurred with recommendations 1, 2, 4, 5, and 6 and described corrective actions the Agency has taken or will take to address them. Those five recommendations are resolved and will be closed upon completion and verification of the proposed corrective actions. Agency officials did not concur with recommendation 3 to quantify overall mission risk ratings and communicate the risks for upcoming launches early, stating that control processes currently in place adequately measure cargo launch risks. We continue to believe that developing a more formal risk categorization approach for cargo resupply missions would better inform Agency management about the risk level of particular missions and foster a discussion about possible ways to mitigate associated risks. As noted in our report, the Orb-3 Independent Review Team mirrored our concerns regarding communication and risk management and made several programmatic recommendations intended to ensure the success of future cargo missions. Furthermore, in our discussions with ISS Program and Office of Safety and Mission Assurance officials, both organizations agreed that a more regimented approach to communicating risk would benefit the ISS Program. Accordingly, we continue to urge NASA to take additional steps to quantify risks for upcoming cargo launches and improve communication of those risks. Therefore, this recommendation is unresolved pending further discussion with Agency officials. The Agency also noted in its response that our assessment of NASA’s unpressurized payload manifesting “oversimplified” the factors that must be considered when determining cargo re-supply payloads and that we “narrowly focused” on cargo launch vehicle risk rather than considering overall risk to the ISS. With regard to payload manifesting, we simply pointed out that in some instances the Agency did not fully utilize the capability and therefore did not receive the best value for its money. We also noted that the Agency has addressed the issue going forward. Regarding the risk issue, we agree an analysis of overall ISS Program risk is essential but continue to believe a quantifiable risk rating for each launch would better inform the overall risk analysis. Management’s full response to our report is reproduced in Appendix E. Technical comments provided by management have also been incorporated, as appropriate. NASA Office of Inspector General IG-16-025 29 Major contributors to this report include, Ridge Bowman, Space Operations Director; Letisha Antone, Project Manager; Kevin Fagedes, Project Manager; Theresa Becker, Team Lead; David Balajthy; Cedric Campbell, Associate Counsel; and Robert Proudfoot. If you have questions about this report or wish to comment on the quality or usefulness of this report, contact Laurence Hawkins, Audit Operations and Quality Assurance Director, at 202-358-1543 or email@example.com. Paul K. Martin Inspector General NASA Office of Inspector General IG-16-025 30 Appendix A APPENDIX A: SCOPE AND METHODOLOGY We performed this audit from August 2015 through June 2016 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. This audit reviewed the SPX-7 failure as it related to ISS upmass capabilities and critical spare parts, SpaceX and NASA corrective actions to return to flight, CRS-1 contract negotiations and modifications, NASA’s risk management processes for CRS-1 launches, and weaknesses for CRS-1 failure investigations. As part of this audit, we reviewed NASA policies and plans, FAA regulations, and USAF procedures. The documents we reviewed included the following: Code of Federal Regulations, Title 14, Chapter III, “Commercial Space Transportation, Federal Aviation Administration, Department Of Transportation” NASA, “Executive Outbrief: F9-020 LSP Independent Investigation,” December 2015 NASA ISS Program, “Contingency Action Plan, Annex A: Commercial Resupply Service (SSP-50190-CRS),” October 2012 and November 2015 versions NASA Policy Directive (NPD) 8610.7D, “Launch Services Risk Mitigation Policy for NASA-Owned and/or NASA-Sponsored Payloads/Missions,” August 27, 2012 NPD 8610.23C, “Launch Vehicle Technical Oversight Policy,” March 6, 2012 NASA Procedural Requirements (NPR) 8621.1B, “NASA Procedural Requirements for Mishap and Close Call Reporting, Investigating and Recordkeeping,” July 15, 2013 NPR 8705.4, “Risk Classification for NASA Payloads,” October 2, 2014 SpaceX CRS-1 contract – December 2008, modifications to the contract, task orders, data requirement descriptions, and work plans SpaceX, “F9-20 CRS-7 Anomaly Report,” November 2015 SpaceX “Post Launch Contingency Plan For the Falcon 9 Launch Vehicle System and Dragon Capsule,” June 2015 To gain an understanding of the impact of the SPX-7 failure, we interviewed officials from NASA’s ISS Program, LSP, and Office of Safety and Mission Assurance. Additionally, we interviewed officials from the FAA Office of Commercial Transportation, USAF, and SpaceX. We also conducted site visits to SpaceX’s headquarters in Hawthorne, California, and Johnson Space Center. We reviewed the ISS Program and CRS-1 contract for organizational weaknesses, risks, and impacts to the Station and ISS Program. We did not review the SPX-7 failure to determine the root cause as this was beyond the scope of our audit. NASA Office of Inspector General IG-16-025 31 Appendix A Use of Computer-Processed Data We used computer-processed data to perform this audit, and that data was used to materially support findings, conclusions, and recommendations. In order to assess the quality and reliability of the data, we verified the information through independent calculations and corroboration with Program documents and the input of various Program officials. Review of Internal Controls We reviewed and assessed controls associated with the audit objectives and concluded that the ISS Program has a comprehensive set of management tools (identified in a detailed Program Plan) that it uses to provide internal controls. To facilitate internal controls, the ISS Program uses a broad set of control boards, panels, and working groups that addresses a myriad of risks. Prior Coverage During the last 5 years, the NASA Office of Inspector General and the Government Accountability Office have issued 11 reports of significant relevance to the subject of this report. Unrestricted reports can be accessed at https://oig.nasa.gov/audits/reports/FY16 and http://www.gao.gov, respectively. NASA Office of Inspector General NASA’s Response to Orbital’s October 2014 Launch Failure: Impacts on Commercial Resupply of the International Space Station (IG-15-023, September 17, 2015) Extending the Operational Life of the International Space Station Until 2024 (IG-14-031, September 18, 2014) NASA’s Use of Space Act Agreements (IG-14-020, June 5, 2014) Space Communications and Navigation: NASA’s Management of the Space Network (IG-14-018, April 29, 2014) NASA’s Management of the Commercial Crew Program (IG-14-001, November 13, 2013) Commercial Cargo: NASA’s Management of Commercial Orbital Transportation Services and ISS Commercial Resupply Contracts (IG-13-016, June 13, 2013) NASA's Challenges to Meeting Cost, Schedule, and Performance Goals (IG-12-021, September 27, 2012) NASA's Challenges Certifying and Acquiring Commercial Crew Transportation Services (IG-11-022, June 30, 2011) NASA Office of Inspector General IG-16-025 32 Appendix A Government Accountability Office Federal Aviation Administration: Commercial Space Launch Industry Developments Present Multiple Challenges (GAO-15-706, August 25, 2015) NASA: Assessments of Selected Large-Scale Projects (GAO-15-320SP, March 26, 2015) Commercial Space Launches: FAA Should Update How It Assesses Federal Liability Risk (GAO-12-899, July 30, 2012) NASA Office of Inspector General IG-16-025 33 Appendix B APPENDIX B: NASA PAYLOAD AND LAUNCH VEHICLE RISK CLASSIFICATIONS Table 8 shows the four payload risk classifications and selected criteria used to determine the classification for a payload. Table 8: NASA Payload Risk Classifications and Selected Criteria Research or Reflight Classifications Priority Cost Launch Constraints Opportunities A High High Critical None B High High to medium Medium Few or none C Medium Medium to low Few Some or few a D Low Low Few to none Significant Source: NPR 8705.4. a Although not officially classified, CRS payloads are considered “like” class D. Table 9 shows the three risk categories and selected criteria used to evaluate a launch vehicle. Table 9: NASA Launch Vehicle Risk Categories and Selected Criteria for Certification Quality Risk Payload Test and Risk Management Flight Experience Systems/ Category Class Verification Management Systems Process No previous flights required, can use the first flight of a common launch vehicle configuration, instrumented to provide Acceptance Test Risk Plan, design verification and flight Plan in place, Mitigated and AS9100 or 1 Da performance data Ground Test, NASA Audit Accepted ISO 9001 (high) End-to-End Tests Technical and Compliant Post-Flight Operations/Anomaly complete Safety Risks Resolution Process Flight Data Assessment Process Alternative 1: 6 consecutive successful flights (89 percent demonstrated reliability) of a Risk Plan, common launch vehicle configuration, Mitigated and 2 C and D, instrumented to provide design AS9100 None None Accepted (medium) sometimes B verification and flight performance data Compliant Technical and Post Flight Operations/Anomaly Safety Risks Resolution Process NASA Flight Margin Verification NASA Office of Inspector General IG-16-025 34 Appendix B Quality Risk Payload Test and Risk Management Flight Experience Systems/ Category Class Verification Management Systems Process Alternative 2: 3 (minimum 2 consecutive) successful flights of a common launch vehicle configuration, instrumented to provide NASA Design design verification and flight Certification performance data Review Post Flight Operations/Anomaly Risk Plan, Resolution Process 2 Mitigated and (medium) C and D, NASA Flight Margin Verification NASA Audit Accepted AS9100 sometimes B Compliant cont. Alternative 3: Technical and Safety Risks 1 successful flight of a common launch vehicle configuration, instrumented to provide design verification and flight Comprehensive performance data Acceptance Test results Post Flight Operations/Anomaly Resolution Process NASA Flight Margin Verification Alternative 1: 14 consecutive successful flights (95 percent demonstrated reliability) of a common launch vehicle configuration, instrumented to provide design None None verification and flight performance data Post Flight Operations/Anomaly Resolution Process NASA Flight Margin Verification Alternative 2: 6 successful flights (minimum 3 consecutive) of a common launch vehicle Risk Plan, configuration, instrumented to provide NASA Design Mitigated and 3 A, B, C, and AS9100 design verification and flight Certification Accepted (low) D performance data Compliant Review Technical and Post Flight Operations/Anomaly Safety Risks Resolution Process NASA Flight Margin Verification NASA Audit Alternative 3: 3 (minimum 2 consecutive) successful flights of a common launch vehicle configuration, instrumented to provide Comprehensive design verification and flight Acceptance Test performance data results Post Flight Operations/Anomaly Resolution Process NASA Flight Margin Verification Source: Summary of NPD 8610.7D. Note: AS – Aerospace Standard. ISO – International Organization for Standardization. a A class D payload can be flown on any of the three classifications of launches. NASA Office of Inspector General IG-16-025 35 Appendix C APPENDIX C: USAF CRITERIA FOR LAUNCH AND PAYLOAD RISK According to Department of Defense policy, the severity category and probability level of the potential for failure for each hazard are assessed using the definitions found in Tables 10 and 11. To determine the appropriate severity category, as defined in Table 10, for a given hazard at a given point in time, the potential for death or injury, environmental impact, or monetary loss is identified. A given hazard may have the potential to affect one or all of these three areas. Table 10: Severity Categories for Risk Severity Description Mishap Result Criteria Category Could result in one or more of the following: death, permanent total disability, Catastrophic 1 irreversible significant environmental impact, or monetary loss equal or exceeding $10 million. Could result in one more of the following: permanent partial disability, injuries, or occupational illness that may result in hospitalization of at least three personnel; Critical 2 reversible significant environmental impact; or monetary loss equal to or exceeding $1 million but less than $10 million. Could result in one or more of the following: injury or occupational illness resulting Marginal 3 in one or more lost work days, reversible moderate environmental impact, or monetary loss equal to or exceeding $100,000 but less than $1 million. Could result in one or more of the following: injury or occupational illness not Negligible 4 resulting in a lost work day, minimal environmental impact, or monetary loss of less than $100,000. Source: Department of Defense Standard Practice: System Safety, MIL-STD-882E, May 11, 2012. To determine the appropriate probability level, as defined in Table 11, for a given hazard at a given point in time, the likelihood of occurrence of a mishap is assessed. Probability level F is used to document cases where the hazard is no longer present. No amount of doctrine, training, warning, caution, or Personal Protective Equipment can move a mishap probability to level F. NASA Office of Inspector General IG-16-025 36 Appendix C Table 11: Probability Levels for Risk Description Level Specific Individual Item Fleet or Inventory Frequent A Likely to occur often in the life of an item Continuously experienced Probable B Will occur several times in the life of an item Will occur frequently Occasional C Likely to occur sometime in the life of an item Will occur several times Unlikely, but can reasonably be Remote D Unlikely, but possible to occur in the life of an item expected to occur So unlikely it can be assumed occurrence may not Improbable E Unlikely to occur, but possible be experienced in the life of an item Incapable of occurrence; this level is used when Incapable of occurrence; this level Eliminated F potential hazards are identified and later is used when potential hazards eliminated are identified and later eliminated Source: Department of Defense Standard Practice: System Safety, MIL-STD-882E, May 11, 2012. When available, the use of appropriate and representative quantitative data that defines frequency or rate of occurrence for the hazard is generally preferable to qualitative analysis. The Improbable level is generally considered to be less than one in a million. In the absence of quantitative frequency or rate data, reliance upon the qualitative text descriptions in Table 10 is necessary and appropriate. Assessed risks are expressed as a Risk Assessment Code (RAC), which is a combination of one severity category and one probability level. For example, a RAC of 1A is the combination of a Catastrophic severity category and a Frequent probability level. Figure 5 assigns a risk level of High, Serious, Medium, or Low for each RAC. Figure 5: Risk Assessment Matrix Combining Severity and Probability Source: Department of Defense Standard Practice: System Safety, MIL-STD-882E, May 11, 2012. Note: The definitions in Tables 10 and 11, and the RACs in this figure shall be used, unless tailored alternative definitions and/or a tailored matrix are formally approved in accordance with Department of Defense policy. The Program documents all numerical definitions of probability used in risk assessments. NASA Office of Inspector General IG-16-025 37 Appendix C Potential risk mitigations shall be identified, and the expected risk reductions of the alternatives shall be estimated and documented in a hazard tracking system. The goal should always be to eliminate the hazard if possible. When a hazard cannot be eliminated, the associated risk should be reduced to the lowest acceptable level within the constraints of cost, schedule, and performance by applying the system safety design order of precedence. The system safety design order of precedence identifies alternative mitigation approaches and lists them in order of decreasing effectiveness. NASA Office of Inspector General IG-16-025 38 Appendix D APPENDIX D: CRS-1 ACCIDENT PLANS AND INVESTIGATIONS Figure 6 outlines the FAA and NASA approval process for CRS-1 accident investigation plans. Figure 6: Approval Process for CRS-1 Accident Investigation Plans Source: NASA OIG analysis of FAA regulations, NASA policies and contracts, and SpaceX’s investigation plans. a Data Requirement Description C1-2, “Mishap Notification, Investigation and Contingency Action Plan,” June 2015. NASA Office of Inspector General IG-16-025 39 Appendix D Table 12 outlines the investigation types, authorities, scopes, and approving organizations during the SPX-7 launch. Table 12: Possible Accident Investigations During SPX-7 Launch Type of Independence Investigation Authority Scope Accident Plan Applied Investigation Requirements Approval FAA Launch License and SpaceX Accident FAA-Required Root Cause No FAA-submitted Accident Investigation Plan Contractor (contractor (not FAA Yes Investigation Plan (after Submitted to the Investigation only)a addressed) delegation by the FAA) FAA (June 2015) CRS contract requirement CRS Contract CRS Contract- Technical (does not reference FAA (Data Requirement No NASA ISS Required Cause accident investigation Deliverable C1-2, (not Program Noc Contractor (contractor authority, but similar in 2008; last updated addressed) Office Investigation only)b application and process) June 2015) Yes NASA Mishap Root Cause Policies and formal Investigation Official NASA Investigation (all Procedures found in NASA Nod requirements Board parties)a NPR 8621.1B and screening CRS Contract (2008) CRS Contract and CRS Contract and ISS Root Cause Informally Yes; Agency ISS Program Office’s Program Office’s (all Similar to NPR NASA No Discretionary Contingency Action Contingency Action parties)a 8621.1B Authority Plan (SSP-50190-CRS (2012) Authority to investigate Technical NASA LSP LSP Informally Yes; failure of same launch Cause (approving Discretionary LSP Contract (2012) similar to NPR Yes vehicle contracted for future (contractor future LSP Authority 8621.1B LSP launch only)b launches) Authority if damage/injury to USAF Contract USAF property or USAF Root Cause (none at time of USAF personnel, or failure of (all launch) or Yes USAF No common vehicle contracted parties)a Air Force Instruction for future USAF launch 91-204 Statutory; Technical Authority to Investigate any Memorandum of NTSB Cause (all Yes NTSB No Catastrophe Understanding with parties)b FAA and USAF Source: NASA OIG analysis of CRS contract and ISS Program policies, FAA regulations and policies, SpaceX plans, NPR 8621.1B, NTSB policies, USAF policies, and LSP launch contracts and policies. a Root cause is the determination of the cause of the failure, including technical, organizational, and programmatic issues, by reviewing the actions by the contractor and all related parties. b Technical cause or proximate cause is condition that directly resulted in the failure (such as a failed strut assembly or engine part). It is usually limited to the contractor’s actions and does not review broader organizational or programmatic issues that may have contributed to the proximate cause. c Because of the FAA investigation, NASA did not require a contractor-led investigation through its CRS contract authority but reserved the right to require additional contractor actions before putting Agency cargo on the next mission. d Because CRS launches are a NASA activity, NASA always has authority to conduct an official mishap investigation but the CRS-1 contract and ISS investigation plan do not require it to occur for every failure. NASA Office of Inspector General IG-16-025 40 Appendix D Figure 7 describes the multiple accident investigation authorities during the SPX-7 mission and how they interact. When the SPX-7 failure occurred, any of these authorities could have been triggered and they could have occurred concurrently depending on the phase of launch and extent of the damage. Figure 7: Accident Investigation Authorities During SPX-7 Launch Source: NASA OIG analysis of CRS-1 contract and ISS Program Office policies, FAA regulations and policies, SpaceX plans, NPR 8621.1B, NTSB policies, USAF policies, and LSP launch contracts and policies. a The Orbiting Carbon Observatory (OCO) was an unsuccessful NASA satellite mission launched on February 24, 2009, and designed to make space-based measurements of carbon dioxide. The Glory spacecraft was an unsuccessful NASA mission launched on March 4, 2011, and intended to collect data on aerosols and measure total solar energy entering Earth’s atmosphere. NASA Office of Inspector General IG-16-025 41 Appendix E APPENDIX E: MANAGEMENT’S COMMENTS NASA Office of Inspector General IG-16-025 42 Appendix E NASA Office of Inspector General IG-16-025 43 Appendix E NASA Office of Inspector General IG-16-025 44 Appendix E NASA Office of Inspector General IG-16-025 45 Appendix E NASA Office of Inspector General IG-16-025 46 Appendix E NASA Office of Inspector General IG-16-025 47 Appendix F APPENDIX F: REPORT DISTRIBUTION National Aeronautics and Space Administration Administrator Deputy Administrator Associate Administrator Deputy Associate Administrator Chief of Staff Executive Officer Chief Engineer Chief Financial Officer Chief, Safety and Mission Assurance General Counsel Associate Administrator, Human Exploration and Operations Director, International Space Station Program Manager, International Space Station Deputy Program Manager, International Space Station Non-NASA Organizations and Individuals Office of Management and Budget Chief, Science and Space Branch Government Accountability Office Director, Office of Acquisition and Sourcing Management Congressional Committees and Subcommittees, Chairman and Ranking Member Senate Committee on Appropriations Subcommittee on Commerce, Justice, Science, and Related Agencies Senate Committee on Commerce, Science, and Transportation Subcommittee on Space, Science, and Competitiveness Senate Committee on Homeland Security and Governmental Affairs House Committee on Appropriations Subcommittee on Commerce, Justice, Science, and Related Agencies House Committee on Oversight and Government Reform Subcommittee on Government Operations House Committee on Science, Space, and Technology Subcommittee on Oversight Subcommittee on Space (Assignment No. A-15-013-00) NASA Office of Inspector General IG-16-025 48
NASA's Response to SpaceX's June 2015 Launch Failure: Impacts on Commercial Resupply of the International Space Station
Published by the National Aeronautics and Space Administration, Office of Inspector General on 2016-06-28.
Below is a raw (and likely hideous) rendition of the original report. (PDF)