Industrial DataOps #7 with Rhize – Geoff Nunan on Why Data Modeling in Manufacturing isn’t optional

Willem (00:00) Welcome, you’re listening to the IT/OT Insider Podcast. I’m your host Willem and in this special series on industrial data ops, I’m joined by my co-author David and Geoff Nunan from Rize. If you want to hear more about what’s shaping the world of industrial data and AI, don’t forget to subscribe to this podcast and our blog. Welcome Geoff. Geoff Nunan (00:20) Well, thanks for having me. Great to be here. Willem (00:23) And, hi David. David (00:25) Thank you Willem. Yes, today we are joined by Geoff all the way from Australia. Geoff is the CTO and co-founder at Rize Manufacturing Data Hub. We’re in for a very interesting episode as we’ll discuss their view on how to work with manufacturing data. Geoff, can you kick things off by introducing yourself and Rize to our audience? Geoff Nunan (00:49) Yeah, thanks, David. Yeah, I’ve been, this is my 30th year in industrial automation and information management this year, 30 years, so showing my age. Most of that time has been spent in MES or MIS, manufacturing information systems, and mostly with larger companies. David (01:01) congrats. Geoff Nunan (01:17) multinationals, the global companies across the whole gamut of manufacturing verticals from mining companies to pharmaceutical and food and bev and right across the range. I have never worked for any of the other big vendors, but I’ve been a systems integrator for most of those 30 years, up until the last seven when I started. to rise and building the product. David (01:50) Cool. that’s more or less the experience of myself and Willem combined. So, we’re lucky to have you on the show. So, seven years ago, what made you start work? Because there had to be a reason why you went like, I can do something better. Geoff Nunan (01:58) Ha ha ha. Yeah, there absolutely was. So we were, I was working for my systems integration company here in Australia. were doing what was a relatively simple MES implementation at a wine bottling plant in the fabulous South Australian wine region that makes Penfolds and other great Australian wines. And we were using I won’t say which product, but one of the big MES products out there at the time. And for something that was a very simple MES, we just want to track downtime. We want to track the consumptions and productions. We want to do some inventory transactions and integrations with ERP. It was an enormously complex implementation and it always is. And we always spend 80 % of our time on 20 % of the requirements because they don’t quite fit the product we’re using to implement them. And so even though you go off the shelf, you end up in software development for a big bunch of that time. And after that project, we were successful with the project. It went in. It achieved the requirements, but it was very, very painful. And it was the last story in many, implementations that were like that with various different products. Really kind of settled in to say, the off the shelf solution is not the right solution. There’s always too much variance in what you need to be able to show a user and no vendor can ever meet 100 % of the variability. that’s in the industry and no vendors ever going to target the cutting edge innovators. They’re always going to target the mass market. And so it’s very difficult to actually push forward. And that’s something in my whole career I’ve been always doing pushing forward towards the front end of what’s possible. So off the shelf. I was convinced by the end of that project that off the shelf was not, I wouldn’t do another one. And bespoke development is not the right approach either for MES, although there’s plenty of that out there. And so I actually started about seven years ago, still as a systems integrator to do manufacturing execution systems using low code. front-end development technologies. We started with a product that’s now global, but at the time it was a Portuguese product called OutSystems. And we built an MES for a global plastics manufacturer, and it was quite successful. And it really proved to me that the front-end is not the hard part of building an MES. The user screens are what you need to do, that always needs to be custom for the company or custom for the operator even. And the backend is really quite consistent across all of the different implementations, can be largely standards driven. And so after that project and a few other initiatives that we were running, we really landed on there isn’t a good technology that allows an open access, open APIs and easy access to an MES data platform, if you like, or even a standardized industrial data platform that you can use the latest low code front end technologies with to go and build manufacturing applications. There wasn’t a headless back end. for manufacturing as there is for content management and as there is for all sorts of other different IT domains have a headless, even banking have headless banking these days, but there wasn’t a headless manufacturing. And so we started really with the vision of being the third option. We’re complete off the shelf. We’re not a bespoke build. We’re a headless backend that works really, really well with low code front end to build applications. But headless backend can also be used for integration and for ontologies and AI as we kind of now know it. That’s a whole domain that’s really developed and accelerated a lot over the last five years, of course. But when we started five to seven years ago, it was really about building this third option for the companies to be able to look at. David (07:46) That’s a lot to digest and think we saw a lot of interesting follow-up questions here. you first of all walk us through the capabilities you need to offer such a platform? I understand that it’s headless, so we’ll talk about… the legs and the torso. What are the different capabilities you need to come to a point where you can indeed attach, I assume not only the visualization tools or the interfaces, but probably also BI tools, training tools, et cetera. But I’ll say what was the torso and the legs. Geoff Nunan (08:16) what that means. Yeah. Yeah. Yeah. Yeah. So it starts with an, what we can call a data model, guess, an information architecture, but, really a data model. and, you know, I, any platform that, you know, putting any SSI, even any IT application that you go and build the hardest part is, is data modeling. It’s. understanding exactly what the use case and the business actually need and modeling that in your information model. We’re very, very fortunate that a large number of very experienced, very capable, intelligent people have come before us and written a lot of that into standards, international standards like ISA95 to actually give us a data model. except they didn’t write the standard with that intent. They wrote the standard, the ISA95 standard, that is, to be a messaging model between ERP and MES, not as a data model standard to put into a headless backend. But we took ISA95 and used it for that purpose. So we start with the data model. Of course, we’ve got to be able to model equipment. We’ve got to actually model the physical process, model our materials and people and the actual processes that we use in manufacturing in that data model. We need to be able to connect to information sources, streams of information from our control systems. We need to be able to logic and workflow in that application. And we need to be able to do a lot of effectively rule processing to say, well, when something happens, then I can take some sort of action. So they’re the kind of big core competencies that need to be in a headless platform. Willem (10:53) I’d like to take one of those out of there, Geoff. Since this is a series mostly focused on data ops, I’d like to have your view on how the maturity of thinking on everything related to data management in manufacturing has been evolving since you started. Geoff Nunan (11:17) Yeah, great question. yeah, we started, I guess, with 25 years experience when we started. And so we’ve our team and the group of senior architects that we brought together to start had 20 plus years of fighting the battles of doing it the wrong way. I would say 20 plus years of doing it the wrong way. And so we started with quite an opinionated perspective on what it’s going to take to actually do it properly. The industry as a whole, I think with data ops in particular has moved and evolved, as you say, quite a lot and continues to every quarter or so. with the maturity of data ops platforms that are available and what you can do and how you can move and manage data. I don’t think they’re all the way there yet, but they’re certainly moving in the right direction. What does it take? What do you need to have? You need to be able to have a robust, reliable system. And that’s not fragile. And what that really means is if I change something in my system, I can know what it’s going to break if I do change that thing. And I can know what I can change that’s compatible with everything else. So can make a change and it won’t break things. What changes I can make in an overall system. that are going to break things. That’s kind of, know, at the root level, that’s what it takes. Achieving that is really, really difficult. So, you know, the way that you don’t do that is to have, say, an MQTT broker with a bunch of topics and have everybody publishing kind of whatever they like into those topics. That’s the way you don’t do it. Because, you know, imagine… You’re a control systems engineer. You’re at one of a hundred different factories and you’re publishing information in. You make a change. You have no idea who’s subscribing to any of the information that you’ve been publishing or what they’ve been doing with it. And you make one little change and the house of cards falls over. So, yeah, it comes through having good standard data structures, having good provenance of where does my data go? Where did it land? Who used it and for what? And if I change it, what will break when I do change it? That’s, think, where the whole industry will eventually end up. I don’t think we’re all the way there yet. David (14:33) Is that what IT data people would call data lineage? Geoff Nunan (14:38) That is, yes, that’s exactly it. Yes. Willem (14:42) Is that a way to scare a bunch of engineers away? I think so. I mean, no, I get what you’re saying, Geoff. How would you explain this to people who are, let’s say, not versed in years of being exposed to the wrong ways of working and how does data management work in industrial environments? When you come to a new client, Geoff Nunan (14:47) Is that the lineage? Yes. Willem (15:10) How you not scared them away on the first meeting? Geoff Nunan (15:15) that’s a really good question. the, the answer to that is, in the context of what are we talking about? So if we’re talking to a control systems engineer at a site who has a local problem, then the things that I’ve just described are not actually a problem because It’s normally one person who has full control over everything that’s being built and how all of that data is going to get used. The system owner at Factory A knows everything about all of the systems there. And so them as a person can kind of keep everything under control. If you’re a multi-site global organization, David (15:51) Yeah. Geoff Nunan (16:09) with perhaps hundreds of control systems engineers, you’re no longer in that environment where one person knows everything about what’s going on and has full control over everything that’s going on. And therefore you need to rely on governance and systems to keep things under control. The thing is though, even in a big company like that, you still have local problems that need local solutions that do not need governance applied to them. The governance just gets in the way of the control systems engineer. So there is absolutely a role for a local ungoverned space where a single person can build a solution to solve a local problem. But there is also… a need and a space for a governed standardized structured data space on which you can build things, on which you can build other applications or teams of people, global teams of people can build applications that are consistent and standardized and can be deployed. David (17:24) I this calls for a practical use case. It’s also when I’m in discussions about data management, the first question I would say I typically get back from the audience is like, but where do we start David? Because… We don’t have too much green fields available. Typically, we’ll start from a brownfield type of approach and maybe there is a new line being built. where do we start? That’s what I’d like to know from you, maybe based on one of your success stories. Geoff Nunan (18:10) Yeah, the answer to that is complex, it’s mostly start where you think the most value is. Start where you think you can win and where you can deliver the value back to your organization. For some of our customers, we have a large customer who’s a luxury watchmaker in Switzerland. And with the price of gold at the moment, really getting great mass balanced level traceability on gold usage across all their different precious metal variations of different carats of gold and different types of gold. That’s where all the value is. But it’s a really, really complex thing to try and do. in a factory to do fine metal mass balance. But just because it’s hard doesn’t mean that that’s not the place to start. The place to start is where the most value is. But you have to build a program and you have to demonstrate that you can get there. So with that customer, as an example, we started with something very, very complex. And then the next thing for them is OEE and performance management, which is normally a much simpler thing to try and do. And other customers have, almost all of our customers have started somewhere different. So I don’t think there’s an answer, a right answer that says you should always start with OEE or water execution or whatever use case. You should start with what are we actually trying to do and why are we doing anything at all? And where do we think we can actually get the return on the money we’re going to spend? Willem (20:21) Is that kind of discussion, let’s say, do you get it mostly initiated through IT departments or do you most often get asked by production themselves or some center of excellence within production? Geoff Nunan (20:39) Yeah, look, we being being quite a technical platform, most people who reach out to us and most of our customers reach out to us and say, please, can we have your platform? Typically coming from architecture or senior people, they’re not we’re not a platform. that is ever sold as we are a solution to the use case because we are a platform rather than an end application. But we’re a platform on which you build many different solutions for many different problems. And so we’re typically a company that people from a technical background reach out to first rather than somebody who is a David (21:13) Yeah. Geoff Nunan (21:36) Let’s say a cost sender owner. David (21:40) I’d like to go a bit deeper in something you mentioned in the beginning. So you mentioned ISA95 as a model. You also mentioned that it’s more about communication than about execution. So I’d love to dig a bit more into this, I would say, what is lacking. Or what did you find lacking in the ISO 95 structure and how did you solve that? And I don’t know whether that’s going to touch the event-driven and the workflow part, but I assume it will. Geoff Nunan (22:17) Yes, yeah, it will for sure. the first thing, yes, ISA 95, right at the start, very first paragraph of the standard, it explains the purpose of the standard, and it is to simplify and standardize the integration between layer four and layer three, or what we would call ERP and MES. That was the purpose of the standard when it was written some 35 years ago. In order to achieve that goal, in order to describe an integration messaging model between ERP and MES, the committee who wrote the standard really had to describe manufacturing as a whole and describe an ontology, a set of concepts to describe this very, very complex space that we are working, which is manufacturing it and all of the different nuances and complexities that we have. And so the standard for anybody who’s tried to read it is nine parts and some thousand pages of documentation. But it is really complex domain, this manufacturing that we’re working, particularly when you start to look across verticals, batch manufacturing, continuous manufacturing, discrete manufacturing, all have their nuances. But the ISA 95 standard is written to be agnostic to vertical. and to really describe the concepts very, very well. So we took that standard and used it as our ontology. So what was missing in that? Well, implementation details. Of course, the standard is a conceptual standard, and there’s implementation detail that’s missing. An example of that is something like versioning. If we model the equipment in a factory, David (24:16) Yeah. Geoff Nunan (24:18) We normally want to keep versions of our models. So yes, we start with version one and maybe we, at some point later, decide we need an extra property in an equipment class. We want to keep what we had before but create a new version. That’s not a problem that a messaging standard needs to really deal with. They can easily just have a field in a type that says version number X. But they don’t really need to handle all of the different history of versions and states of versions and all these other things. That’s an example of something that isn’t in the ISA95 standard that we’ve really had to look and implement a solution for in the ontology. David (25:11) interesting because that actually touches one of the problems we faced multiple times is you have this interface between your MES and your ERP system at a certain point in time you’re to make a change and it actually requires well there is not really a test system typically so that requires to have this this coordinated action between putting the change in production on your MES side and your ERP side and then hoping just hoping the thing still works Geoff Nunan (25:35) hope it. Yes. Yeah. Yeah. And that’s something fortunately that the ISA95 standard helps a lot with having that standard model in between and being able to isolate changes to some extent between different systems. We take that and implement it as the ontology in a data platform. that all of your applications can connect to and write their information into. So taking it a step further than just being a message model, an information model for data in flight, but making it an information model for data at rest in a database. So all of the information in that database is all standardized, is all related to each other in a full ontology. And it’s all clean and ready to actually use for analysis and other things. Because it’s in a standard structure, every implementation, every database is exactly the same. Regardless, if you have 100 factories, the schema of the information is exactly the same at every factory. That’s where you get the value. Willem (27:01) When you mentioned previous question that you were mostly interacting with IT teams and architecture teams. And again, you’re not there to solve the problem of an individual automation engineer in a single plant. That implies that there’s also some sort of OT counterparts or production counterpart that’s mature or well developed or organized enough to have those. requirements across plants and have a discussion with IT before they even get you. So in your experience, how does this look like in practice? How does this interaction between IT, which is of course a central organization, but OT, which in essence starts with being the central. Geoff Nunan (27:53) Yes, and often continues to be decentralised because, well, everything’s fairly locally specific. There’s very few companies that actually have a lot of centralised Ooty because you’re dealing with machines at a factory that are quite specific. Yes, they might have centralised procurement and some standards and things like that, but… David (28:17) Yeah. Geoff Nunan (28:21) mostly fairly decentralized. Where does it start? To be honest, it starts with companies failing multiple times to try and do what they want to do. And I say that with all sincerity. All of our customers that we work with have failed at least four or five times with big implementations, trying to move the needle, trying to get to get digital transformation done over many, years. So sometimes they’ll have been trying to do this for 10, 15, 20 years. We work with a very large steel industry customer in Australia. That’ll narrow it down because there’s only one of them. David (29:16) you Geoff Nunan (29:18) They had tried and failed four times to implement the ISA 95 standard within their organization previously. And so the difficulty is real. And you learn every time you fail, you learn. But also, and we’ve just successfully rolled out to four. sites with a new MES for them, replacing a system that was first implemented in 1985. So it’s the green screens with the text user interface. We’ve just taken one of those out. But the reason that that system had lasted so long is because it’s really hard. David (29:55) 1985. Geoff Nunan (30:13) you know, replacing big systems in big organizations is really hard. And so we’ve managed to replace that and roll from site to site for sites in four months with that organization of rolling out a new system. yeah, it’s kind of sad to say, but most people get to the point where they realize they need to have a centralized, standardized approach after they’ve failed multiple times. I guess there’s an obvious lesson in that for those that haven’t failed multiple times yet. Perhaps speak to some of the people who have and learn some of the lessons. But for us, that’s where it comes from. David (31:11) fact that you worked with multiple companies who failed means that there has to be some kind of a common thread here. Is it because they are unable to model the actual process and they then try to implement workarounds and those implement in those workarounds they become, I don’t know, they become bigger and bigger until a point it stops working or what would be the main reason. Geoff Nunan (31:42) So partially, would say, I mean, you can assemble a great team and have a great result in one or two or three sites. And I’ve recently been discussing with a large European dairy company who did this after many trials before. had a successful MES. pilot and project and rolled to a few sites and really had a great team and a large team to make sure those first few sites were successful. But then when it came to rolling out to the other 150 factories, I realized the economics don’t work. It’s too costly. It’s too complex. We’re not going to be able to deploy this out to all the other sites. And so it’s back to the drawing board. It’s a new team, new people, new ideas, and you start again. And I’ve seen over 30 years that I’ve been doing it, companies go around in that circle time and time again, lose the team, lose the knowledge, new people come in, start again, go through the same cycle, end up in the same place, start again. So for us, one of the keys to solving that is working with the standard, because at least then you’re standing on the shoulders of others. You’re starting with a data model that’s been quite well proven for 30 years. It’s also, we’ve proven that we can take graduate level software engineers and train them in that standard and in manufacturing in four to six months. which greatly reduces the amount of time it takes to build an MES implementation team. Part of the challenge for most companies is it takes 20 years to build a good MES engineer. So if we could shorten that to six months and build teams of people who can actually implement standardized well-structured systems, then that partly solves the problem. And the third part is flexibility of the technology. It’s can we have the all of the things standardized that can be, but still the flexibility to do whichever use case we want, whichever factory we want with a screen that looks exactly the way we want. And then it’s about speed of iteration and innovation. And can we keep that moving and sustain that over time? Willem (34:40) What I’m hearing you say is in a sense that companies fail start to recognize that to be able to scale, need to think differently about their data and that it takes lots of iterations to fail. Now, lately, David and I have been hearing a lot from people with a lot of questions more often than answers when it comes to UNS, which in essence, think, touches upon this. How do you see this changing the way that companies will fail? And what gaps do you still need to fill up? Geoff Nunan (35:17) Yeah, that’s a great question because it is. It is an example of the same cycle happening. So you see, even over the last few years, lots of people coming in, hearing about UNS for the first time, think, OK, I just need to try it. I can get started quickly and inexpensively. And they do. And it’s great. They learn. And they learn a lot and learn a lot quickly. And that is a good thing. The trouble is that in most cases, they’re learning too slowly and too painfully and they’re not being shown the tricks, I guess, if you like, or the things to watch out for that won’t trip them over now, but will in a year or two years time as they get there. Willem (36:18) What kind of things are those? Are those more technical, organizational, process? Geoff Nunan (36:25) Yeah, it’s data model. It’s back to data model every time. So if you’re designing something and you’re designing the data model for it and you’re designing that data model based on all the requirements you can see in two years time, you’ll be able to see a completely different set of requirements. And what happens every time is you discover a new requirement that doesn’t fit with the data model that you’ve built over the last two years. And then you have to go back, throw it all away and start again. And so, yeah, the thing that I would love more people to learn and understand is that you don’t have to start with a clean sheet of paper on the data model. There is a data model out there. It’s ISA 95. And it will take you and it will feel painful at the start. And the reason that it will feel painful at the start is because your sphere of requirements are only this big and the standard was designed for a set that are way bigger than that. But you will get to the rest of them in a year or two years or five years time. Particularly if you can iterate quickly, you’ll get through the rest of those. But if you follow the standard, it will help you and it will guide you and it will stop you from tripping over your own feet as you go. So that’s the biggest thing for me. Willem (38:06) I have to say I’m also hearing different approaches to this data modeling exercise. There is one view which is indeed let’s make one very complete model. The other one is let’s make more localized functional domain, use case even data models. What’s your take on the balance between those two? Geoff Nunan (38:33) Local models for local problems will work just fine. So if you’re the only person who’s going to use it, think about Excel. How much do you trust anybody else’s Excel spreadsheet? Yeah. And so if you’re building something and you’re designing it and you’re the only person who’s really ever going to own it and support it and have to update it when it changes, do whatever you like. Honestly, that’s where a lot of data ops tools that are in the market will suit you just fine. And you’ll be able to do what you need to do. And it will take you quite a long way. You will get to a point where you’ll trip over and you won’t be able to keep going because you’ve built the wrong data model. But if it’s a local problem that doesn’t ever grow that far and you’re happy for it to be a silo, that’s not integrated into everything else, then it’s not a problem. Have a local solution for a local problem for a local person. If you’re building a multi-site MES or if you’re building an application which you would one day like to be able to deploy to another site within your organization, or if you’d ever like to be able to use something that another site in your organization has spent the money building, and you’d like to be able to pick it up and copy it and use it in your site. That doesn’t work. The local data models, local designs don’t work anymore. You’ll never be able to copy and paste somebody else’s solution into your plant if you don’t have a common foundation and a common data model underpinning it. David (40:25) I think this is a perfect ending of this episode where we explored how to make industrial data work for us, but really with a big emphasis on data modeling. Big thank you to you, Geoff, for sharing your insights and to our listeners for tuning in. If you enjoyed the conversation, don’t forget to subscribe at itotinsider.com and leave us a rating. and see you next time for more insights on bridging IT and OT and until then, take care, bye bye.