February 11, 2017

Screen Testing preview: Deepwater Horizon

When I saw Deepwater Horizon in the cinema a few months ago, one thing struck me immediately about the first hour of the film:

Now that the film's available to buy/rent, I decided to check whether my first impressions still held true. They resoundingly did - so much that they've formed the basis of a new podcast which I'll be co-hosting, launching in March! Screen Testing will look at popular films and TV series which feature software testing elements (either explicit or implicit) and discuss how true-to-life they are.

There'll be an introductory episode in the next few weeks, but in the meantime, if you're thinking "I'm sure a film about a disaster on an oil rig can't have that many testing elements", let me show you just a few!

(Don't worry, no spoilers.)

Tight deadlines encourage bad decisions

Deepwater Horizon - 43 days behind schedule

Many of the issues on the Deepwater Horizon were an indirect result of a late-running project. The importance of the schedule was at the forefront of everyone's minds, with a wall-mounted sign drawing everybody's attention to the financial imperative to stay on-track.

The slipping schedule was used as the justification for skipping much of the usual due diligence on a drilling project. For example, after pouring a protective cement layer around the line, the concrete was usually left for a day or more to set, and then multiple tests ("positive" and "negative") would be performed to ensure the concrete's integrity met agreed levels. As Mark Wahlberg's character arrives on Deepwater Horizon, the test team is departing on the same helicopter - they'd run the positive tests, but BP had advised them to skip the negative tests to save time (as one character notes, "I guess they [BP] assumed that would just be OK"). BP manager Donald Vidrine (a typically phenomenal John Malkovich performance) declares "We are confident in the integrity of our cement job", but it's blind confidence based on partial information.

There's one particularly memorable scene where rig supervisor Jimmy Harrell takes his BP supervisors to task over the decision to skip these tests. He tells Vidrine:

"You don't want to know if that cement job is shit because you're 43 days and $50million over budget. You sent the testing team home before they could do their job, and maybe give you some news you didn't want to hear."

Harrell points out that the cost of running this extra test (and the subsequent further delay to the project) was miniscule for a company operating on BP's scale: "You're a $180bn company and you're cheap!"

No slack to fix technical debt

Deepwater Horizon - blue screen of death

The tight schedule also meant that important day-to-day tasks were being sidelined in favour of meeting targets, and even this was only being achieved with overtime and double-shifts. Mark Wahlberg's character, chief technician Mike Williams, gives a conservative estimate that 10% of all the vessel's systems were in need of maintenance and repair, including critical support systems such as the phone system, the air conditioning and even the toilets.

Williams presents this information to BT in an attempt to demonstrate that the team needs to take a step back to address these problems, but instead Vidrine asks Williams to precisely calculate the safe operating margins for optimal (and cheapest) running. Williams hits back -

"Running out of gas as the plane lands isn't smart, it's flawed thinking. It's hope as a tactic."

This happens throughout software development too - it's my experience that addressing technical debt is often sidelined in further of building new features on top of a shaky framework, and that activities such as "10% time" is the first to go when a perceived critical deadline is approaching.

Poor communication across reporting lines

Deepwater Horizon - Vidrine whiteboarding

The tight schedules were clearly creating opportunities for conflict and disagreement, but it was obvious that the crew didn't feel comfortable in preventing their viewpoint to managers, not least because their managers seemed to give them the brush-off. (As one worker shrugs to a colleague after a seemingly baffling decision, "They never feel the urge to take me through their thinking".)

The chain of command itself seemed fundamentally flawed; the film is littered with dialog like this (when discussing the progress of the cement tests):

A: "I heard we're done."
B: "Who told you that?"
A: "I can't tell you."

Similarly, answers to difficult questions were often vague or non-committal ("...not really"), deferred to somebody else ("You'll have to talk to Vidrine about that") or bypassed altogether ("Don't ask me, I just drive the boat", "It's above my pay grade").

These erratic communication strands are indicative of a systemic problem in the heirarchy on the Deepwater Horizon, and it was evident that BP recognised it and actually used it to their advantage. While trying to force the crew to make a tough decision, Vidrine declares: "You're nervous as cats. That will be my summary back to my bosses."

Interpreting the results of tests

Deepwater Horizon - Running tests

When BP are eventually persuaded to allow the negative pressure test, the initial run of the test reveals abnormally high pressure (in true movie style, a gauge is seen going up into the red, so we know it's bad). As sirens sound to indicate a high pressure alert, the crew declare this to be a "bad test" (i.e. a failure), but the BP response is "Not necessarily". They argue that if this was a "legitimate pressure issue", there would be other symptoms - for instance, a failure of this type would normally be accompanied by mud leaking out of the system, but no mud was seen. Both sides had their merit - the problem itself was the flaky test whose outcome could not be trusted.

After discussion, the teams agree to repeat the same test on the secondary line (the "kill line") to see whether the results can be reproduced. The test on this line returns green (zero pressure), which the BP executives take as a sign that drilling is ready to begin. This time, it's the crew that objects - they point out that the two lines are analogous and should therefore be returning the same reading, meaning that further investigation was required. BP dismisses this however, due to the lack of confidence in the first reading. So for a second time in a few minutes, we have the result of a test - this time, a supposed successful test - where the outcome isn't obviously indicative of the system's state.

The perfect storm

Deepwater Horizon - Warning light

Even when a catastrophic leak occurs, it takes time for the rig workers to work out what's happened. We see an illuminated dashboard with a red sensor, and the crew's first reaction is "Gas alarm going off in the mud pits, again. Same one that goes off every time."

In other words, all of the seemingly small things that led to this moment - time constraints, malfunctions, pressure from management and a lack of confidence in monitoring systems - built to allow a catastrophe to occur. And while my discussion about the film stops at this point, there's still half of the film left, and it's as compelling a disaster movie as I've seen in quite a while.

As the film concludes with testimony and interviews with the real-life survivors (and victims) of the tragedy, a voiceover tells us something that we should all bear in mind when we're discussing seemingly trivial concerns in our projects: "All the things that they told us could never happen, happened. Men lost their lives."

Stay tuned for more news on Screen Testing, launching in March. Follow us on Twitter (@ScreenTesting) for the latest!,

  • LinkedIn
  • Tumblr
  • Reddit
  • Google+
  • Pinterest
  • Pocket
Comments powered by Disqus