U.S. C H E M I C AL S A F E T Y A N D H A Z A R D I N V E S T I G A T I O N B O A R D

INVESTIGATION REPORT VOLUME 2 EXPLOSION AND FIRE AT THE MACONDO WELL (11 Fatalities, 17 Injured, and Serious Environmental Damage)

DEEPWATER HORIZON RIG MISSISSIPPI CANYON BLOCK #252, GULF OF MEXICO

KEY ISSUES IN VOLUME 2 APRIL 20, 2010

• BOP TECHNICAL FAILURE ANALYSIS • BARRIER MANAGEMENT AT MACONDO • SAFETY CRITICAL ELEMENTS

REPORT NO. 2010-10-I-OS 6/5/2014

[This page left intentionally blank.]

Volume 2 Technical findings on the Deepwater Horizon blowout preventer (BOP) with an emphasis on the effective management of safety critical elements

[This page left intentionally blank.]

Contents VOLUME 2 .................................................................................................................................................. 3

ACRONYMS AND ABBREVIATIONS ................................................................................................... 10 1.1 Volume 2 Synopsis ........................................................................................................................ 13 1.2 Key Findings .................................................................................................................................. 14

2.0 CONTROLLING FORMATION PRESSURES WITH THE DEEPWATER HORIZON BLOWOUT PREVENTER ........................................................................................................... 17 2.1 BOP Sealing Elements ................................................................................................................... 19 2.2 The BOP as a Physical Barrier....................................................................................................... 21 2.3 Functioning the Deepwater Horizon BOP ..................................................................................... 23 2.3.1 BOP Control System......................................................................................................... 23 2.3.1.1 Functioning Solenoid Operated Valves ......................................................... 25 2.3.2 BOP: Closing the Blind Shear Ram .................................................................................. 27 2.3.3 Initiating the AMF/Deadman Sequence............................................................................ 27 2.4 Condition of the Well on April 20, 2010—Data Used to Recreate the Incident Events................ 28 2.5 The Macondo Well Kicks—Incident Analysis of Well Control Response.................................... 28

3.0 THE BLOWOUT PREVENTER – FAILURE OF A BARRIER .................................................. 31 3.1 Correlating Physical Evidence from Macondo with the Events of April 20, 2010........................ 32 3.2 Failure Analysis of the Deepwater Horizon BOP .......................................................................... 34 3.2.1.1 Blue Pod: Disconnected Wires and the Drained Battery............................... 36 3.2.1.2 Yellow Pod: Miswired High-Pressure Shear Closes Solenoid ...................... 37 3.2.1.3 Successful AMF/Deadman Tests on the Yellow Pod.................................... 38 3.2.1.4 Independent CSB Exemplar Solenoid Testing .............................................. 38 3.2.2 The AMF/deadman Successfully Fires on April 20, 2010................................................ 39 3.2.3 The AMF/deadman Fails to Seal the Well: Buckled Drillpipe ......................................... 43 3.3 Conclusion ..................................................................................................................................... 45

4.0 ESTABLISHING AND MAINTAINING EFFECTIVE BARRIERS .......................................... 47 4.1 Defining the Role of a Barrier: Major Accident Events ................................................................ 47 4.2 Barriers to Prevent or Mitigate MAEs ........................................................................................... 49 4.2.1 Visualizing Barriers using a Bowtie Diagram .................................................................. 52 4.2.2 Determining the Type and Number of Barriers to Reduce Risk ....................................... 55

4.2.3 Maintaining Effective Barriers ......................................................................................... 57 4.2.3.1 Barriers as Safety Critical Elements (SCEs) ................................................. 58 4.3 Conclusion ..................................................................................................................................... 61

5.0 DEEPWATER HORIZON BOP NOT TREATED AS A SAFETY CRITICAL ELEMENT ...... 62 5.1 Identification of a SCE................................................................................................................... 64 5.1.1 BOP Component Failure Identified in DWH Hazard Analysis ........................................ 64 5.1.2 DWH Hazard Analysis Did Not Address BOP Design Capabilities ................................ 65 5.2 Defining Performance Requirements of a SCE ............................................................................. 66 5.2.1 Drillpipe Exceeded Shearing Capabilities of DWH Blowout Preventer .......................... 66 5.2.2 Prescribing Minimum Reliability Requirements of a BOP .............................................. 68 5.3 Performance Assurance of an SCE ................................................................................................ 70 5.3.1 No Assurance Activities for the Critical AMF/Deadman Solenoid Valve ....................... 71 5.3.2 Current Deadman System Function Tests Are Inadequate ............................................... 72 5.3.3 Assurance Activities of Human Actions ........................................................................... 76 5.4 Gap Closure ................................................................................................................................... 77 5.5 Verification Activities—The Independent Competent Person ...................................................... 78 5.6 Conclusion ..................................................................................................................................... 79

6.0 ANALYSIS OF RECOMMENDED PRACTICES AND REGULATIONS REGARDING THE BOP AND OTHER SAFETY CRITICAL DEVICES .................................................................. 81 6.1 Lifecycle of SCEs under BSEE ..................................................................................................... 82 6.1.1 Hazard Analysis not Focused on Targeted Risk Reduction of Major Accident Events ... 82 6.1.1.1 Lack of Targeted Risk Reduction Requirements: Parallel Findings between the CSB Investigations...................................................................................................... 84 6.1.2 Lack of Defined Performance Standards for all SCEs...................................................... 85 6.1.3 Performance Assurance and Verification Needed for all SCEs ........................................ 86 6.1.4 Gap Closure Important for Continuous Improvement of SCE Effectiveness ................... 87 6.2 Regulatory Responses Post-Macondo: Prescriptive Change versus Continuous Improvement .... 88 6.2.1 BOP Shearing Capability—An Illustrative Example of Diverse Regulatory Responses . 89 6.2.2 Proposed Regulatory Changes Suggest US Recognition of the Importance of Lifecycle Management of Safety Critical Equipment......................................................................................... 92

7.0 VOLUME 2 CONCLUSIONS: TECHNICAL SAFETY FAILURES REVEAL BROADER REGULATORY GAPS ................................................................................................................. 93

8.0 RECOMMENDATIONS ............................................................................................................... 95

APPENDIX 2-A: DEEPWATER HORIZON BLOWOUT PREVENTER FAILURE ANALYSIS ......... 98

APPENDIX 2-B: DEEPWATER HORIZON RBS 8D BOP MUX CONTROL SYSTEM REPORT ...... 99

APPENDIX 2-C: SCENARIOS WHEN TWO BSRS WOULD NOT BE OPTIMAL ............................ 100

REFERENCES ......................................................................................................................................... 101

Figures and Tables

Figures Figure 2-1. The DWH BOP stack ............................................................................................................... 18 Figure 2-2. An annular preventer can seal the annular space around a drillpipe or an open hole. Pistons press up on the rubber component which pushes it inward to seal around the pipe or open hole. ......................................................................................................................................... 19 Figure 2-3. A pipe ram can seal the annular space around a drillpipe, but not an open hole without drillpipe present. ...................................................................................................................... 20 Figure 2-4.Control panel (left) and partial closeup of control panel on the Deepwater Horizon found in the driller’s cabin and on the bridge of the rig. These controls are used to activate the BOP....... 23 Figure 2-5. Pressing a pushbutton on a BOP control panel sent an electronic signal through the MUX cable down to the yellow and blue BOP control pods located in the LMRP. Accumulators on the BOP stack supplied hydraulic power to the control pods during emergencies. ................ 24 Figure 2-6. The top image depicts a solenoid with no current running through it. The plunger is down, and no fluid can flow through the solenoid. When actuated, current running through the solenoid produces a magnetic field which creates a force that pulls the plunger up, allowing fluid to flow.......................................................................................................................................... 25 Figure 2-7. Simplified schematic of the control pod battery arrangement.................................................. 26 Figure 2-8. Key operation events after reservoir flow began...................................................................... 29 Figure 3-1. (Left) Photograph of Y103 wire arrangement from Phase II testing with pins 1 and 4 connected to white wires and 2 and 3 connected to black wires. (Right) Schematic of correct arrangement of wires, with pins 1 and 3 connected to white wires and 2 and 4 connected to black wires............................................................................................................................... 37 Figure 3-2. Miswiring in the blue pod caused the critical 27-volt battery to drain, rendering the pod inoperable during the incident. A drained 9-volt battery in the yellow pod left one of the coils in the miswired Y103 solenoid valve inoperable, allowing the other coil to activate unopposed and initiate closure of the blind shear ram. ........................................................... 40 Figure 3-3. The events that led to the likely partial closure of the BSR after the emergency AMF/deadman system activated on April 20. .................................................................................................. 42 Figure 3-4. The Deepwater Horizon BOP was designed to shear centered drillpipe (left) in the BSR and then seal the well. During the Phase I examination of the BOP, the drillpipe was found off- center (right), causing the BSR to close only partially, leaving the well unsealed. ................ 43 Figure 3-5: Theoretically straight pipe with equal inside and outside pressure (left); real pipe with a curve imperfection with equal internal and external pressure (center); pipe buckling as a result of increased internal pressure (right). The black wedges show the relative change in length and area of the two sides of the pipe. ............................................................................................. 45

Figure 4-1. Hierarchy of Controls. .............................................................................................................. 51 Figure 4-2. Bowtie diagram depicting the relationships between hazards, barriers, and the major accident events they are intended to prevent. ........................................................................................ 53 Figure 4-3. Bowtie diagram showing potential decay mechanisms of the technical barriers intended to prevent a fault during temporary abandonment activities. ...................................................... 54 Figure 5-1. Simplified representation of the management system for the lifecycle of a safety critical element .................................................................................................................................... 63 Figure 5-2. Simplified schematic of the Cameron FAT procedure to test the AMF/deadman. .................. 75

Tables Table 2-1. Various components of a BOP and their uses, (See Appendix 2-A for model numbers and capabilities of the DWH BOP elements) .................................................................................. 21 Table 3-1. In addition to the three phases of DWH BOP testing from May 2010 to April 2011, the CSB completed independent exemplar solenoid valve testing in September 2012. .......................... 35 Table 4-1. Excerpts from offshore regulations from the UK, Norway, and Australia that specifically require Major Accident Events be addressed; they are juxtaposed with US regulations that promote safety and environmental protection, but without a focus on MAEs. ......................... 49 Table 4-2. Recreated excerpts of Transocean's Risk Assessment for the DWH ......................................... 50 Table 5-1. Recreated excerpt of Transocean's MAHRA for the Deepwater Horizon ................................. 65 Table 5-2. Summary of emails sent between Transocean personnel regarding BSR shearing capability. . 68 Table 6-1. Excerpts from offshore regulations from the UK, Norway, and Australia concerning a required analysis., .................................................................................................................................... 83

Acronyms and Abbreviations

ALARP As Low As Reasonably Practicable AMF Automatic Mode Function API American Petroleum Institute BOEM Bureau of Ocean Energy Management (United States) BOEMRE Bureau of Ocean Energy Management, Regulation, and Enforcement (United States); the US offshore safety regulator between June 18 and October 1, 2011a BOP Blowout Preventer BSEE Bureau of Safety and Environmental Enforcement (United States); US offshore safety regulator since October 1, 2011 b BSR Blind Shear Ram CCPS Center for Chemical Process Safety CSB U.S. Chemical Safety Board CSR Casing Shear Ram DNV Det Norske Veritas DOI Department of Interior (United States) DOSH Division of Occupational Safety and Health DWH Deepwater Horizon EDS Emergency Disconnect System GoM Gulf of Mexico HSE Health Safety Executive (United Kingdom) LCM Loss Circulation Material LMRP Lower Marine Riser Package LOWC Loss of Well Containment MAHRA Major Accident Hazard Risk Assessment

a Department of Interior, Order No. 3302, Change of the Name of the Minerals Management Service to the Bureau of Ocean Energy Management, Regulation, and Enforcement (June 18, 2011), http://www.doi.gov/deepwaterhorizon/loader.cfm?csModule=security/getfile&PageID=35872. Accessed February 19, 2014. b The Reorganization of the former MMS, http://www.bsee.gov/About-BSEE/BSEE- History/Reorganization/Reorganization/. Accessed February 19, 2014.

MGS Mud-Gas Separator MAE Major Accident Event MMS Minerals Management Service (United States); US offshore safety regulator at the time of the Macondo accident until June 18, 2011 a MODU Mobile Offshore Drilling Unit NOPSA National Offshore Petroleum Safety Authority (Australia) NOPSEMA National Offshore Petroleum Safety and Environmental Management Authority (Australia, successor to NOPSA) NTL Notice to Lessee OCS Outer Continental Shelf POSC Presidential Oil Spill Commission PSM Process Safety Management PETU portable electronic test unit PLC programmable logic controllers ppg pounds per gallon PSA Petroleum Safety Authority (Norway) psi pounds per square inch SCE safety critical element SEM subsea electronic module SEMS Safety and Environmental Management System SPPE Safety and Pollution Protection Equipment UK United Kingdom US United States USCG United States Coast Guard VBR Variable Bore Ram

a Department of Interior, Order No. 3302, Change of the Name of the Minerals Management Service to the Bureau of Ocean Energy Management, Regulation, and Enforcement (June 18, 2011), http://www.doi.gov/deepwaterhorizon/loader.cfm?csModule=security/getfile&PageID=35872. Accessed February 19, 2014.

Volume 2 – Approach to Analysis Volume 2 Overview Macondo is an international problem whose lessons extend Chapter 1 – The focus of this volume and beyond the United States. The global business of offshore the key investigation findings that support exploration and production continues to advance in the CSB analysis. complexity. Meanwhile, the catastrophic consequences of Chapter 2 – The sealing capabilities of a another incident on par with Macondo threaten not only the BOP as a physical barrier and the incident welfare of the workforce, public, and environment, but the events pertaining to the DWH BOP’s industry’s long-term viability. The international nature of integrity at the time of the incident. this business allows for all stakeholders to learn from each Chapter 3 –The CSB failure analysis of the DWH BOP, and the implications for BOPs other—many companies operating offshore do so on a used offshore. global level. Companies can bring their individual best Chapter 4 – Concepts underlying technical, practices wherever they go; the equipment, facilities, and organizational and operational barriers for people used to conduct offshore operations travel between major accident prevention. regions as needed; and regulators worldwide have Chapter 5 – The lifecycle of a safety recognized the need to disseminate knowledge through critical element and deficiencies in the information sharing forums. a treatment of Deepwater Horizon BOP emergencies systems. No one offshore region operates within a framework that Chapter 6 – Recommended practices and provides an undisputed panacea to prevent all accidents. regulations pre- and post-incident for the Challenges and undiscovered hazards exist in every offshore BOP and other safety critical elements. location. For example, within this volume, the CSB has Chapter 7 –Major conclusions to illustrate identified a key weakness in BOP function testing important lessons for industry and the US promulgated in internationally accepted industry guidance. regulator.

Regulatory regimes can only provide the foundation for Chapter 8 – Recommendations for industry and the US regulator. effective major accident hazard management, and failures by any one company to carry out the intent of the regulatory requirements may occur in any offshore region. Yet a foundation is essential for ensuring that all those operating offshore are reducing risk to a level acceptable to themselves, the regulator, and society as a whole. Examining the strengths and weakness of the various major accident prevention approaches used by industry and the regulator—both in the US and elsewhere—can identify and improve attributes that provide for more effective safety management. This is a primary aim of the CSB’s overall investigation into the Macondo incident and the focus of this volume.

a Some examples include the International Regulators’ Forum (http://www.irfoffshoresafety.com/) and the North Sea Offshore Authorities Forum (http://www.ptil.no/nsoaf/category999.html; http://www.ens.dk/en/oil-gas/health- safety/international-cooperation-2/north-sea-offshore-authorities-forum).

The CSB provides its failure analysis of the BOP to spark a global reexamination of how industry is managing safety critical elements a as well as regulatory requirements and approaches used to ensure that these management practices are effective.

1.1 Volume 2 Synopsis b The Macondo well blowout began when the Deepwater Horizon (DWH) crew was in the final stages of temporarily abandoning the well so that a production facility could return later to extract oil and gas. BP’s temporary abandonment plan c called for removing the upper portion of the drilling mud in the well before installing a surface cement plug. d The decision proved fateful because both BP and Transocean personnel on the DWH rig had misinterpreted test results e concerning the cement integrity at the bottom of the well. This error led the personnel to believe that the hydrocarbon bearing zone at the bottom of the well had been sealed when it was not. Ultimately, the blowout preventer (BOP) was the only physical barrier that could have potentially contained well fluids, but only if the crew or emergency systems could have successfully engaged it. f As the events of April 20, 2010 indicate, the BOP did not seal the well. In analyzing the BOP failure to seal the well during the incident, Volume 2 of the CSB Macondo Incident Investigation report has five objectives: 1. To discuss key preventable hardware shortcomings affecting the reliability of the Deepwater Horizon BOP throughout the drilling activities at Macondo. 2. To account for all conditions that can cause drillpipe to buckle in a well, leaving it off-center in a BOP and potentially interfering with the BOP’s ability to seal a well. These conditions include having buckled drillpipe even when a rig crew has successfully shut in a well. 3. To explore safeguards, or barriers, that help prevent major accidents, recognizing they extend beyond physical equipment into operational and organizational elements. 4. To describe the necessity for effective identification and management of safety critical elements—technical, organizational, and operational—for preventing Macondo-like events.

a Safety critical elements are controls (hardware, people systems, or software) or tasks whose failure could cause or contribute to a major accident event or whose purpose is to prevent or limit the effects of a major accident event. (See Section 4.2.3.1) b See Volume 1 for a basic introduction to deepwater drilling and physical barriers that can prevent a blowout. c A well may be sealed temporarily with cement or mechanical plugs to allow removal of the blowout preventer and departure from the drilling rig. d Cement plugs are portions of cement put into a wellbore to seal it. “Surface” is typically used to refer to the most shallow cement plug used in a well. e A number of human and organizational factors contributed to how the events unfolded leading to accepting the test results. The CSB plans to address these factors in Volume 4 of the CSB’s Macondo Investigation Report. f Well integrity also includes the casing lining the wellbore, float valves (check valves) placed at the bottom of the casing, and crossovers where casing of different sizes are connected to one another. Analysis in Appendix 2-A indicates the major source of hydrocarbons during the incident did not come from casing or crossover failures. While check valves can act as a physical barrier, they are unreliable and cannot be independently tested. For the analysis in this report, they are not considered a barrier because at Macondo they were either not converted or had to have failed.

5. To identify additional opportunities for improvement in the US offshore safety regulations that do not include clear and systematic requirements to ensure the successful performance of all safety critical elements (SCE) for reducing major accident events.

1.2 Key Findings The redundant controls of Deepwater Horizon BOP should have increased the reliability of the BOP to seal the Macondo well during normal drilling operations and emergency situations. Two rounds of post- incident testing, including one non-public, court-ordered round and additional CSB testing, reveal new failure mechanisms in which these redundant controls can be compromised and go on undetected. From this analysis and an examination of how the BOP, was managed and regulated as a safety critical element, the following key findings demonstrate the need for further offshore safety improvements: BOP Failure in Loss of Well Control 1. The BOP is subject to design capability limitations. A BOP can act as a barrier only if it is closed manually by the drilling crew or automatically as a result of a catastrophic event, such as a fire and explosion, which can trigger emergency backup systems. In manual operations, successful closure