Agile – How Digital Gets Done

76932EE7 E51C 4A5D 82DA D6875C3871A4 e1531490114747

Agile – How Digital Gets Done

One word that does not describe the oil and gas world is “agile”. Why is being agile suddenly something that the industry aspires to?

The WaterFall Method

When I started my career with Imperial Oil in Toronto (yes, long ago, Toronto was home to the industry), I was taught that there was only one way to deliver computer technologies for users. It was called the waterfall method. IT analysts interrogate unwilling and confused users to understand what they need by way of automation using imprecise language and quirky documentation methods. These requirements are thoroughly hammered out before being handed to software developers to start coding, a process that could take months or even years.

Eventually, the coders emerge and present their creation to the startled user (now a different person after such a long gestation), who says something along the lines of “that’s not what I wanted”, or “that’s nice but the business has changed”.

This reaction kicks off fresh and protracted efforts to remediate the software by salvaging as much of it as possible. God hope that there wasn’t a contractor involved, because such problems often lead to lawsuits. Every few months there is some media story about a $100m software project that has not worked out, with recriminations and litigation.

The waterfall process did solve for many of the legacy approaches to the technologies of the past. It made sense to deliver detailed specifications to the coders so as to FIX THE SCOPE of the work to be done, which would minimize the demands on coders, who were relatively scarce and expensive. Fixed scope in theory would drive cost accuracy and schedule reliability, provided ALL the requirements were accurately captured by the analysts.

It wasn’t just the software developers who were highly paid and in short supply. Hardware was typically a costly mainframe computer with a dumb terminal. Mainframes had limited capacity, were expensive and slow. Networks and disk storage were equally costly and rationed carefully.

Waterfall as a method works reasonably well where business processes are highly regulated, users have limited flexibility in how they work, and business processes change infrequently, if at all.

But the waterfall method creates its own hazards. The process is front-end loaded, and poorly defined requirements create lots of risk down the track. Users of technology don’t often know what they want and so the requirements can be wildly inaccurate, incomplete, miscaptured, poorly detailed, or worse. All the feedback comes at the end of the process, leaving few ways to correct the direction of the project if it’s heading astray.

The waterfall approach is not well suited to projects where the user doesn’t know precisely what they want, or where the user lacks experience with automation.

It will not be lost on an asset intensive industry like oil and gas that the design and build of processing assets like gas plants and wells is also done using a waterfall approach. It’s called stage gate, but it’s the same method. Requirements are hammered out in pre FEED (or front end engineering and design), then sharpened in FEED before being released to build and construction.

The difference is that the protocols for capturing engineering requirements are much more mature. There are standards for things like steel, and the performance features of assets (weight, throughput, energy consumption), are well documented. The different engineering disciplines have their own standards to follow and their respective documentation styles, which also contributes to what the industry calls interface issues (where an electrical panel needs to be attached to a physical structure creates an interface between the electrical team and the structural steel team).

Managers in oil and gas rightfully wonder why waterfall, which works reasonably well in engineering, does not work so well with technology?

The Agile Approach

Compare this to agile technology development. Built for an age with virtually free computing, networking and data storage, agile FIXES THE TIME available to deliver some functionality, rather than the scope, thus creating the opportunity for more frequent directional feedback. Instead of analysts working with users to agree the scope to minimize coding, combined teams of analysts, users and software developers work collaboratively to deliver a working solution quickly, recognizing that some coding work will be throw-away. Teams work in a series of sprints to refine their understanding of the business problem, and work transparently together rather than in silos to deliver a minimally viable product or MVP.

The interactions between IT and users are transformed, leading to happier users and better solutions. Value, in the form of working functionality, is delivered frequently, in smaller chunks, and not at the end in a large single release.

An important nuance to agile is the release schedule. New versions of software are released much more frequently than the once-in-six-months model that I was familiar with at Imperial Oil, and not all at the end as with a commissioned gas plant. The coders must work with the team that maintains IT hardware, such as servers and networks, at the same pace that analysts, users and coders collaborate to design the minimally viable product.

It makes no sense to have a fast-moving agile team generating multiple small iterations of a product or solution every few weeks, only to have their efforts hit a slow moving operations team that can only release software every 6 months.

Amazon deploys new software 50 million times per year. You can see this with the regularity of releases of software on your smart phone. That just cannot be done manually. Agile techniques must also apply to the team that handles overall releases, carries out regression testing, automates user tests and integration, and executes the actual release and deployment.

Executives ask me why it is that oil and gas companies feel so slow when compared to digital companies, and a big part of the explanation is the use of agile to drive iterative change.  

Successful digital teams in oil and gas use agile methods for solution delivery. Chances are very high that users don’t understand how digital innovations could change their business, and digital professionals don’t understand oil and gas. Agile teams overcome this limitation. I would have thought that users, having only experience in waterfall methods, would reject agile out of hand, but not so. Users welcome a chance to work with a new solution developed in just a few short weeks, rather than waiting months for a more complete but incorrect solution.

Agile Teams

People issues feature prominently in the world of agile delivery. CIOs have figured out that they need to create the role of Product Manager on agile teams to drive the accountability and decision making focus for new digital solutions. The Product Manager is embedded in the team, and has to have significant involvement – it’s not a part time casual job. Teams have to be multi-disciplinary and cross functional, which means collaboration with other managers to find and free up the right people for the job.

Not everyone is suited to work in an agile way – it requires participants to be able to unlearn the legacy way by which they get their work done, which could be a tall order for someone who has worked in the same way for a very long time.

To build momentum using agile on digital projects, managers seed new digital initiatives with the experienced users, analysts and product managers from previous projects. Managers start projects small (so that value can be quickly delivered), and aim to scale projects up once they reach a high level of value release.

Managers have figured out that carefully selecting the team members is key – great team members have to like project work, be ready and able to embrace new ways of working, adapt well with uncertainty, are influential with their peers, and can personally grow to take on bigger roles on a digital transformation.

Finally, the best of the best see agile as the way to create an agile business culture. Instead of confining agile to purely digital technology projects, industry pacesetters use agile in process areas to innovate processes at the same pace as technology.

For more insights on how agile is being used in other industries, check out this blog article.


Check out my book, ‘Bits, Bytes, and Barrels: The Digital Transformation of Oil and Gas’, available on Amazon and other on-line bookshops.

Take Digital Oil and Gas, the one-day on-line digital oil and gas awareness course.

Mobile: ☎️ +1(587)830-6900
email: 📧
website: 🖥
LinkedIn: 🔵

  • Pingback:Agile – How Digital Gets Done – Oil Industry Job Updates
    Posted at 06:25h, 11 June Reply

    […] post Agile – How Digital Gets Done appeared first on Digital Oil and […]

  • Michael McCormack
    Posted at 17:44h, 11 June Reply

    Excellent analysis and overview of both methodologies and their impacts.

    • geoffrey cann
      Posted at 20:15h, 11 June Reply

      Michael – thanks for the feedback.

  • Ray Simonson
    Posted at 06:05h, 16 June Reply

    Geoffrey. Great article. I remember those waterfall software projects. I’m worried that the energy industry is still too much in the hands of the gray haired guys to accept your proposition.

    • geoffrey cann
      Posted at 14:04h, 16 June Reply

      Ray – for the most part, I think you’re right. There are a few oil and gas companies who are actively driving agile delivery, so all is not lost.

Post A Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.