30 Mar Interview With Mark Fiala of ProductionAR
This is a transcript from a podcast I recorded with Mark Fiala, Founder and CEO of ProductionAR, about the use of augmented reality in industrial settings to improve the productivity and safety of workers.
Geoffrey: Welcome back to Digital Oil and Gas. My name is Geoffrey Cann, and I’m the host of this podcast. Today, I’m joined by Dr. Mark Fiala who’s the founder of ProductionAR, a very interesting company based in Edmonton.
Mark: Thank you very much, Geoffrey.
Geoffrey: Let’s dig a little bit into your background. What’s your life story, that got you to a point where you’re now a technology entrepreneur?
Mark: I’m a computer vision scientist gone bad, one that wants to build a company and make things that work in the real world, as well as do the background science that makes it possible. I’ve been an electronics engineer, a researcher at a federal lab, a prof, and a lifetime builder and futurist. I believe computer vision is the future that’s going to enable a lot of the things we see in sci-fi, but I really believe in the solid, practical, fundamental pieces that actually work. I’ve spent my life working in that direction.
Geoffrey: Sounds like you share an enthusiasm for visual technology with Elon Musk, who also believes that computer vision technology can be truly superhuman. Many of his cars are actually using visual systems, not LIDAR and other technologies.
Mark: Yes, I’ve long thought about the technology that we see in sci fi movies and how to make them become reality. What technology or technical pieces do we need? As a teenager building various robot projects, I concluded computer vision, that is the ability for machines to see, is really the fundamental building block for self-driving cars as you described, and augmented reality, as well. But it’s interesting; computer vision is a term that for many decades, I had to explain what it even was— the idea that a machine is seeing and processing video for some automated final purpose and not just change the image to then present to human eyes. I have a lifelong passion for this. I realized computer vision is a necessary piece to build all the cool things that we see in sci-fi movies.
Geoffrey: You’re founder of ProductionAR. How long has ProductionAR been in action? What you’ve been working on has been many years in development. Unlike an overnight success, it actually started a decade ago.
Mark: It’s been out there for probably about 15 years, this technology. Initially it was somewhat successful in the academic world, and many projects were using it. For instance, there’s thousands of ants that have spent their entire lives with markers glued on their back in Switzerland, in a government lab. Now, I am building it into a commercial success with solid building block and modules. So, to answer your question, the actual corporate entity is called Millennium3 Technologies Inc, a holding company for anything to do with this technology. But the brand name is ProductionAR, which was rolled out about two years ago, and the corporation has been around since 2011. And the technology has been licensed well before that to industry, mostly in Europe and also in the film industry in Hollywood. ProductionAR is in use in a logistics application in a factory here in Edmonton, and two other companies, one warehouse and another one that does some remote inspections. That’s under the brand name ProductionAR.
Geoffrey: We’ve touched on a number of interesting use cases for the technology, but let’s step back and talk through the business problem that you’re using the technology to actually solve. Walk me through a scenario where your concepts of using computer visualization technology actually comes to life.
Mark: ProductionAR is geared towards augmented reality. The AR in ProductionAR is for augmented reality. And that’s when we bring information from the computer world and put it in the real world.
So, imagine a large Canadian energy company with a number of field assets. A lot of the information they need about the assets are already in some system, in some server room somewhere. Workers print out documents they need, and then drive 50-70 miles to get to a site. Now they’re sitting in a warm Ford F150 truck, because it’s minus 30 Celsius here in the Canadian winter. They have to begrudgingly climb out of the vehicle, climb across some snow, maybe climb up some ladder to get at the asset. And then, four out of five of the documents are correct, but one of them is the wrong document.
With augmented reality, you aim your phone at something and information pops up hovering over it. You tap on it, and it takes you to more information. That’s the vision of augmented reality— pull out your phone, aim it at something, and if there’s any kind of information that’s useful, such as manuals, maintenance logs, real-time information, temperature, pressure, warrantee notes, remote assistance perhaps to some expert at the home base. All that could come up. The dream is for this to come up immediately on your phone.
Imagine you’re driving along past a bunch of pump houses and pipes and other assets, and you’d like to have some information about them. Imagine if we could just get instant access to it without having to go through some sort of complex process. We have a beautiful digital wireless network, but we just don’t have all this data pulled together.
Until Elon Musk perfects the neural lace, one of his other many ideas, which hopes to connect our brains directly to computers, we really just have this smartphone in our pocket.
Geoffrey: …On your phone or tablet or whatever. Ok, I imagine that I’m in my truck 60 feet away from the pump house and it’s my job is to go and inspect it. So, using my phone, what do I do? I point the phone and the camera at the pump house. What am I looking for?
Mark: Our system uses QR code-like markers. Picture QR codes, but on steroids.
Geoffrey: The little black matrix for scanning? You see them on products, magazines?
Mark: They’re black and white, 2-dimensional images for a barcode scanner. And often companies have them almost as a status symbol, a graphic element you could have on your web page or your poster.
Geoffrey: I have one of my business cards. If you click on the QR code on my business card using an iPhone, it takes you directly to my website.
Mark: Exactly. Imagine if that QR code was about five and half inches wide, and it’s on that pump house that you’re sitting 60 or 70 feet away from. Then, with my iPhone 6S, I can see it clearly from 75 feet away. Imagine that in your field of view, there might be several such markers. There’ll be little icons hovering over each. And then whichever is closest to the center of your screen, some text will appear underneath it, and perhaps a PDF icon. And if the asset is also pushing real-time information, it might show a little, blinking, yellow caution symbol, as a warning. Or a little speedometer dial that’s showing you the pressure. Or a little PDF icon that you can tap and now you can put your phone down—you’re no longer in the augmented reality mode—but you’re scrolling up and down on your phone, looking at a manual that is the instructions that are associated to that pump house.
Geoffrey: Right. I imagine being in a warehouse where you have dozens and dozens of these things, and you can literally point your phone, roll it around the warehouse till you find the item you’re looking for, with the phone is reading these QR codes on steroids, and pulling up information about whatever happens to be tied into the corporate system.
Mark: Our factory customer has a stainless steel or anodized aluminum handle, and they have human-readable text and numbers laser etched in. And they also have our little marker in the corner, maybe a couple centimeters wide, a centimeter high. With their phone or tablet, they look at it, and they use it to track and scan different items through stages of production. To bring up information for the final quality control check, they use an iPad Pro8 with a tough book case-cover, look at it, and bring up a PDF. They click on it, and it brings up the original drawing that this custom item should be.
I think there’s a lot of interesting stuff happening in the computer vision space. But practically speaking, it’s on our phones for now. And then there’s wearable devices, kind of like a Google Glass for industry. Wearable Google Glass light devices that clip onto your hardhat are coming soon. Imagine you’re driving your forklift around your warehouse, your hands are free, you don’t have your phone in your hand, you just see little icons and arrows blinking. Fairly simple graphics that help the person zoom around quickly and get to where they should be and move the box to where it should go. I think it’s going to be a pretty interesting world, and a very visual world. I’m a very visual person. I just picture all these beautiful graphics over things. We’re adding information to the real world.
Geoffrey: This highlights the shortcomings of using just GPS data. Another way industry could try to achieve the same thing is to pull up information about what asset should be located that a GPS location. If your job is to, say, inspect a flare stack or a water separation unit or a control panel associated with a SCADA system, the GPS data is just not granular enough, it lacks the fidelity to be able to go down to the individual asset level.
Mark: Good point, Geoffrey. For example, Pokemon Go is an example of GPS plus the phone’s estimate of what orientation it’s facing. When you have a cute fantasy character, it doesn’t matter if it’s actually drawn 90 degrees off from where it actually is, because it’s just for fun.
We use computer vision as the alternate sensor, working with the camera on the front of your phone, and that’s what’s recognizing the objects. That way we can put arrows and information directly over things like in the movies. Augmented reality is Hollywood special effects but in real time. So, you have your SCADA system, and you have various registers and readings you want to bring up, but you have many devices all around you. GPS and the phone’s location is approximate, but you can use computer vision for precision. Now we can have dozens of things all in a tight, small space, and little arrows and graphics that line up perfectly with the actual object. That is just not possible with just the GPS location alone.
Geoffrey: You mentioned that there are a few companies using this technology. What are the benefits that they are experiencing by this visualization technologies? Is it happier employees? Is it time savings? Is it lower error rates?
Mark: Well, different people have different uses. We’re starting at commercialization and spreading it out more into the industry. We’ve had two years, so we’ve got three different cases and one is controlling chaos. Chaos may hard to define financially. But if you imagine…
Geoffrey: We’re in the world of coronavirus and oil price collapse. I think the oil industry is pretty hip on chaos.
Mark: I guess oil fields are always a little chaotic, good and bad. But imagine a business where things appear to be running smoothly based on your plan. For instance, we need to earn 30% margin. And then if it’s a tight industry, the margin’s less and less. Everything has to work really smoothly. Supplies have to arrive just so. Work has to get done, there can’t really be any confusion, because anytime things go slightly wrong, you start losing money and you’re struggling.
So the cost of chaos is hard to define.
With this factory, they overproduced, they made the wrong things, they make all these custom items, they would lose things, their order was supposed to be 25 and they sent 17. After a while, unfortunately, they get blamed for things that weren’t even their fault. So, they held hundreds of thousands of dollars of extra inventory.
Then, they brought in an AR system. Their employees have inexpensive Android devices spread out through the factory, the phones in people’s pockets. Now they’re not losing anything. So, they control chaos, and they give more confidence for their customer, because the customer can also log in, and see some limited info on order-progress. If something went missing, they have an electronic paper trail, now they’re not losing track of production.
Another customer does inspections of remote facilities. They’re using it to bring up information about assets, and also to quickly ascribe locations to things, because it’s just an app using a metal etch or sticker. They are using it to make a business dashboard at the home base. They’ve got a couple of big smart screens, showing different graphics, and showing the same data on their phones. The manager has a dashboard. Their benefit is the home base dashboard.
Another customer sells insulation products. We’re rolling this out fairly soon, but the idea is this. Imagine you’re a site manager, at a large oil facility. You’ve stored 150 trailers in a yard, but nobody’s really sure what’s in any of them. But imagine if you and the contractors can all pull out your phones, aim at the trailer markers, and use augmented reality with your phone. Now you see what’s in all of the trucks. And, when things were loaded at the warehouse, the system knew not only what was in the truck, but what skid positions within the trailer. Augmented reality is a way to pull it all together, so we don’t lose that info. We pull out our phone, we pull out our tablet, we look at something and if there’s any useful information, we get it right away.
Geoffrey: I can anticipate a large laydown yard of spare parts and materials and equipment. There’s a need to know what’s available and where. Imagine flying a drone over a large lay down facility. If these QR code symbols were pointed skyward, the drone would be in a position to very quickly tell you where something is located. That could be very, very handy, particularly on turnarounds, because in a turnaround situation, time is money. There’s a lot of equipment brought to bear to execute the turnaround. And frequently companies will buy or rent two or three versions of the same thing, just because it’ll go lost and they need it urgently on the turnaround. Imagine the ability to find these things very, very quickly.
Mark: With the current slowdown and economic pressures, companies that already have outstanding contracts must finish the work in a much more painful economic environment. Some of these industries that have been supplying the oil sector are suffering losses. Before, they would invoice the oil company, who just accepted it. There wasn’t really much transparency or accountability. The system we’re rolling out for the insulation company, for example, is something they can offer their customers. The trailer leaves, and we have all the information there. The contractor takes the truck in and out, and takes inventory out. So now we know how much has been taken out by who, we can build up a running invoice in the cloud. Real-time transparency and accountability.
Geoffrey: That’s an interesting angle too. What you’re describing is the possibility of sending an inventory of some products or goods to a site, and the customers are only billed when they consume the products as opposed to buying the whole container load. Because if they’re buying the container load, it’s no longer the supplier’s responsibility, the customer takes title. But if you could find a way to allow the parts supplier to own that inventory, and control it tightly, then the customer could potentially pay only for what they consume, as opposed to what is in the container. That’d be really interesting shift in economics, enabled by digital for supply chains.
Mark: As well, we still have pristine things that were in the trailer. Let’s say we took out some insulating material, then we found out we only used half of it. We’ve got the flows set up so that the contractor can put it back in that container. Now they’ve returned it partially used, or partially consumed. Perhaps the contractor still has paid for it. And next time, the contractor needs a certain 45-degree, three-inch PVC elbow or something, they first take the ones that they’ve already bought, they can store it in the same trailer. When you look at it with your phone or a tablet on the inside of a wall that you can see with augmented reality, what the status of something is. You take the stuff that you’ve already purchased that belongs to your contracting company, and at the end of the day, you don’t actually take it out again. But then it goes back in the container, back to the warehouse, and then there’s a restocking fee and a used fee for the material that can be reused. The contractor saves money too, because they can return part of it, and we have a nice history. You can look at something and bring up a history of those items. That saves everybody money.
Geoffrey: It feels to me like augmented reality would drive a fair amount of potential process change inside companies. That often creates challenges during deployments and rollouts. What have you seen by way of resistance to these ideas and technologies?
Mark: Well, in some cases, I believe there may not be much resistance other than the people are just too busy. While it would be really nice if they could bring up this information so they didn’t drive for an hour and print out the wrong document, some people are just so busy doing their current job, they can’t even stop to consider change. Anything computer-ish they start to shrink away from.
It’s really important to find decision makers, because so far, one resistance point has been the workers. But usually when they see the demo they’re, “Wow, that’s really cool.”
Whoever makes augmented reality for industry products, you have to make it so it smoothly goes in. You should only give the minimal amount of information necessary. You can’t add more work to the person, the workers. It has to save them work. So that’s been a major obstacle so far. It’s really just: “I don’t want to take on new, additional work.”
As for the solution? I have a nice story from Mark Twain, in his autobiography, when he was a riverboat captain. They were trying to form a union of pilots to drive the river boats. The riverboat companies weren’t interested. The insurance company ended up being the ones that drove this change, because armed with the records and the skills of approved pilots, they suffered far fewer riverboat crashes. In the same way, in our situation, even if there is a little bit of a resistance, as soon as insurance companies or high level managers who worry about the bottom line get involved, then I think it’ll change.
One other obstacle is that people often think a certain way about QR codes, so they need to see a demo in person. To get them to look at a video or a live demo, it takes a bit of work to get to that point, unless you happen to see them at a trade show. People just have the vision of QR codes and barcodes in their mind. And they just imagine it being sort of the same thing.
Geoffrey: Barcodes are unavoidable if you do any kind of shopping at all. I don’t know a single cash register anymore where the clerk actually types in the price. They scan the barcode. We’re all used to that as an idea. The problem is that you have to be six inches away. So the mental model that people have is that “I have to be right on top of this thing in order for this to work,” and what you’ve created, or what this technology does is it gives you a huge spatial distance between you and the physical thing.
There’s a significant safety element there, which I think could be very appealing to oil and gas. Keep people out of harm’s way. They don’t have to get physically close to things unless they absolutely have to. That will be very useful in a business world of physical distancing.
Mark: Absolutely. Keeping people safe from coronavirus is one idea. Raising safety practices is another example too. One electrical inspection company wants a system to give them a little icon of what type of clothing you should be wearing when scanning equipment. For certain high-power electrical equipment, if you’re within 30 feet, you should be wearing certain gear. They already have written standardized signage for safety, but they also want to be able to bring up some additional info with phones or headsets.
One of the ways people resist these changes is they just imagine these apps and computer systems that you have to go through all these various steps to access. And then they forget their login, password, or how to access certain information? It’s like the old days of VCRs: “oh my goodness, what mode am I in now?” But the nice thing about augmented reality is that the context is not set by navigating through menus. You look at something, you aim your phone at something, and information just appears. You don’t have to go through any menus. You don’t have to adjust any settings.
In the bigger picture, I think augmented reality’s time is here, because everything’s becoming so connected. We have the cloud technology. We have wireless data. We have so much data, it is just unfortunately not in the right place at the right time, and it’s not easy to get access to.
Also, I think of augmented reality as inverse IoT. Today, we have IoT, Internet of Things. Various companies are producing sensors to bring more data from out in your facilities back to your server room. It’s information; more and more information from in various buildings and equipment. We want more and more going back to home base, with stuff like SCADA systems. Augmented reality is the opposite: we want information from our central system, whether it be in the cloud or our private server, in our main building. Let’s do the opposite to the IoT, let’s pull that information back out into the world. Information going in the other direction.
Geoffrey: A big change in mindset and thinking about this. That’s really interesting insight. Mark, this has been a great discussion. Thank you very much for sharing your perspectives and innovation today. If someone wanted to learn more about this, where do they find you? What’s your website address?
Mark: The website is ProductionAR.com. Our main product at the moment is called TagScan. You see a link for that on the ProductionAR webpage. That’s the logistics and information and manuals system we have.
Geoffrey: Thank you very much Mark. It’s been a great discussion about computer visual technology and how we can better use it in the real-world setting.
Mark: Thank you very much, Geoffrey. I’ve enjoyed it as well.
This transcript has been edited for readability.
Check out my book, ‘Bits, Bytes, and Barrels: The Digital Transformation of Oil and Gas’, available on Amazon and other on-line bookshops.
Take Digital Oil and Gas, the one-day on-line digital oil and gas awareness course.
Mobile: ☎️ +1(587)830-6900
email: 📧 firstname.lastname@example.org
website: 🖥 geoffreycann.com
LinkedIn: 🔵 www.linkedin.com/in/training-digital-oil-gas