Value Driven Data Science: Boost your impact. Earn what you’re worth. Rewrite your career algorithm.

[00:00:00] Dr Genevieve Hayes: Hello and welcome to Value Driven Data Science, the podcast that helps data scientists transform from technical implementers to strategic experts by contributing to the decisions that really matter. I'm Dr. Genevieve Hayes, and today I'm joined by Serg Masis. Serg is the principal AI scientist at Syngenta.
[00:00:23] A leading agricultural company with a mission to improve global food security. He is also the author of Interpretable Machine Learning with Python and co-author of the upcoming DIY AI and building responsible AI with Python. In this episode, you'll discover practical strategies for building interpretable machine learning models that will earn stakeholder trust and accelerate AI adoption within your organization.
[00:00:53] So get ready to boost your impact, shape decisions, and own your expertise. Serg, welcome to the show.
[00:01:01] Serg Masis: Thank you for having.
[00:01:02] Dr Genevieve Hayes: I first encountered interpretable machine learning when I was working in insurance, although I didn't realize it at the time. The insurer I worked for used machine learning models as part of its premium calculation process.
[00:01:16] And there was an unwritten rule that any models we deployed had to be easily explainable to our policy holders. My employer was a government insurer that sold compulsory work cover insurance to businesses throughout the state. And the motivation behind this rule was simple. For people to have confidence in the work cover system, the calculation process needed to be transparent as a result.
[00:01:40] Model interpretability was implicit in everything my team did. That was more than a decade ago. The stakes have only gotten higher, and while AI and machine learning seem to be increasingly heading in the black box direction, the need for interpretability has never been more critical.
[00:01:57] Particularly in high stakes fields like medicine, law, and government, where algorithmic decisions can fundamentally alter the course of someone's life. Serg, you've literally written the book on machine learning interpretability, and given it's now in its second edition, I'm guessing the topic has resonated strongly with many data scientists.
[00:02:18] What originally motivated you to write interpretable machine learning with Python?
[00:02:23] Serg Masis: I became interested in the topic a few years earlier, back when I had a startup. And I had a different mindset than I was applying machine learning to my project, but from a very engineering standpoint it was very naive and I imagined I could take any part of my project apart.
[00:02:45] Understand it holistically. To me the model is just like any other component in the machinery. And so I encountered a problem the needed of a solution. Something was going wrong with my models and I wasn't sure why. At that point I didn't even know it was the models. So I just started to debug the code and then it arrived at those models and I hit a wall. 'cause I had no idea how to debug them. It seemed quite simple. I had trained them. I had the data, could possibly have nothing wrong. And I wasn't sure. Yeah, that got me down that rabbit hole, of trying to discover what made models.
[00:03:25] Do what models do, and why do they make the decisions they make and what makes something of more value to a model than others? Questions like that, and it's a fascinating world once you get into it.
[00:03:40] Dr Genevieve Hayes: What sort of techniques were you using to begin with when you were first trying to debug these models? Were there techniques that you'd researched or ones that you developed yourself?
[00:03:49] Serg Masis: No, there were techniques I saw in academic journals because there was very little written on the subject. In fact, I don't know if I had quite arrived to the terms that are widely used right now. 'cause now people say explainable ai, interpreter machine learning. Back then I don't even remember what the terms were, but they weren't as obvious as that.
[00:04:09] And they weren't debugging for sure. That's not a term that was used. I stumbled upon some academic journals and so I pretty much used some local interpretability techniques. And with those I could see what was going on with individual instances but I left unsatisfied because you wanna understand a model holistically too.
[00:04:30] You wanna understand, not only what's going on with a few instances, but, maybe a collection or all of them. That got me motivated to learn more about it and learn how to apply it to different kinds of models as well.
[00:04:43] Dr Genevieve Hayes: When I first started going down the interpretable machine learning path, I didn't even realize that was the name for it, and it didn't even occur to me that there might be something written on it. So a lot of the techniques that I was using were just things that I'd. Made up myself because I made sense, like holding all of the variables constant except for one and just seeing what would happen and it actually did.
[00:05:09] Tell me a lot about how the model worked under the hood.
[00:05:13] Serg Masis: Yeah. Yeah. And in fact, one of the first chapters in the book go into that, you can visualize it. Assuming you have not too many features you can visualize, a few of them at a time, three dimensions where you hit a limit there visualization wise.
[00:05:30] Of course you can do other kinds of techniques to visualize it, you can do some dimensionality reduction or something like that. But, I really enjoy visualizing that, how does the decision boundary look like in a model? Why is it in the place where it is?
[00:05:49] To me that's fascinating thing that, you step over the line and then it's a completely different decision and it seems arbitrary. 'cause what is it for the two points that are. On the other side of the line. It's like they're entirely in different countries.
[00:06:05] They cross a line and go somewhere else. It's just arbitrary, I think sometimes because it doesn't seem natural. But that's the way a lot of models operate,
[00:06:14] Dr Genevieve Hayes: I used to teach a machine learning class to universities. Students, and one of the things I'd have them do is visualize the decision boundaries for their models when they're learning about these different types of models and some of the decision boundaries you can get for things like SVMs and the like there.
[00:06:33] Ridiculous. Have you seen that picture of the spiral decision boundary?
[00:06:39] Serg Masis: Yeah. That's. Crazy. Yeah. Yeah.
[00:06:41] Dr Genevieve Hayes: I don't even know how they managed to fit that one.
[00:06:46] Serg Masis: yeah, that's a interesting one. As we end.
[00:06:49] Dr Genevieve Hayes: Two terms that come up again and again in your book are interpretable and explainable before reading your book. I use those terms interchangeably, but you draw a distinction between them. Can you explain to our listeners what that distinction is?
[00:07:06] Serg Masis: Yeah, it gets confusing because explainable AI became the term that was co-opted to discuss. Or just explainable machine learning, anything that was quote unquote black box. In other words, deep learning models all sorts of models that I don't think are nearly as, they're not black boxes in the typical sense, because you can definitely look at the weights if you have access to them.
[00:07:34] You can look at, their inner workings. Can you understand it? That's a whole different story. On the other hand, you have what is quote unquote white box models, and they're assumed to be something that's clearly holistically explainable because there's a mathematical formula you can disentangle and everything.
[00:07:54] But there are limits to that because, you could definitely fit a linear regression model with, 2000 variables. Can you understand it? That's a whole different story, or you could also fit a decision tree that's a hundred layers deep, that doesn't necessarily mean it's interpretable
[00:08:12] so I think the distinction done by industry that, you call interpreter machine learning. Everything that is white box, quote unquote, like simpler models and everything that is black box is given. Explainable AI is silly. I would argue it's actually makes sense for it to be opposite.
[00:08:33] That's something I argue in the book because to me something that's explainable has a much higher bar than something that's interpretable. Because if you can explain something is because you can understand it more holistically than something you're interpreting.
[00:08:48] Something that you're interpreting, you're already admitting that there's some level of subjectivity to it. It's your interpretation, someone else can have a different interpretation explanation. You could also argue the same thing to some degree, but to me it's a higher bar.
[00:09:03] It just seems semantically interpretation is something that is subjective. By definition and I wouldn't necessarily argue that's necessarily a bad thing because if you're a professional and you know what you're doing, your professional opinion must have some value.
[00:09:20] On the other hand explaining something you have a deeper understanding what's going on. That's more like a semantic reinterpretation of what those two terms mean in those contexts.
[00:09:30] Dr Genevieve Hayes: Okay. I think I understand what you're saying. So to give an example back when I was working in insurance one of the models that came onto my desk for me to sign off on. Was a generalized linear model, which had three and possibly even four way interactions in it now, because that's a linear model that would've been interpretable because you can very clearly see what goes on under the hood, but those three and four way interactions were impossible for me to explain to policy holders. So even though it was interpretable, it wasn't explainable, is that what you're saying?
[00:10:08] Serg Masis: Yes. Yes. Because to me, if you could explain it, you could explain it's inner workings. But interpretability is to me more like a question of post hoc. There is a term that kind of predates a lot of these things. It's one of the first terms I learned about, which was post hoc interpretability.
[00:10:27] There had already been quite a few conferences, the FA conferences, which were academic conferences about this topic. And I think they started doing those in like 2012 or 2013. And there's fairness, accountability, and transparency and the topic kind of fell under that umbrella of fat.
[00:10:46] FAT. And those conferences would discuss the topic of post hoc interpretability and post co interpretability is what we typically do with the black box models. Since we can't really dissect the underlying formulas behind it and say, oh, it's doing this in this layer, in the deep neural network in this weight is responsible for this.
[00:11:10] It's really hard to attribute these things. Post hoc interpretability is not about understanding the formula before things are happening, but. After, so you're just throwing things at it. Usually you're using some kind of permutation or something like that on the inputs to figure out what's going on in the outputs.
[00:11:29] So that's how you figure out what went on. You're not necessarily going into the guts of the machine to figure out what is going on there. It's more of a question that once you've done that you can already do things like attribution to the features or other things with it.
[00:11:46] The post hoc interpretability, I think makes a lot of sense and I wouldn't diminish its value because like anything that's complicated, that's what we're doing anyway. Like the human brain is complicated and we're doing post hoc interpretability, on our decisions.
[00:12:00] We're not necessarily figuring out exactly what neuron did what at what time, so I think it's fine. For a while it had a bad rap in the sense that it wasn't the same thing as having this holistic understanding of the models in our workings, which I think is of course very valuable.
[00:12:20] But sometimes, I would say from a responsible AI point of view, responsible machine learning practice, it makes absolutely no sense to opt for a lesser performing model. Simply because you're gonna have worse outcomes, and that's not necessarily the responsible outcome that you want.
[00:12:41] So there's obviously a trade off going on, which is discussed in the book.
[00:12:46] Dr Genevieve Hayes: I could imagine a scenario, for example, in medicine where. You want the models to be interpretable, but you wouldn't want to have a lower level of accuracy 'cause that could lead to diseases not being treated. And so I could imagine that'd be a scenario where you might wanna opt for a black box model, such as some sort of deep learning model, but then use interpretability techniques.
[00:13:11] Serg Masis: Yeah, definitely and there's an awful amount of creativity that's needed it's not just domain knowledge, but it's also thinking a little bit out of the box and understanding that there is a possibility of having a hybrid approach. You could use a relatively simple decision tree to figure out, okay, how do I categorize these patients?
[00:13:33] High risk, low risk, et cetera. So you can understand that. And based on their level of risk you figure out whether you wanna use like more of a scalpel kind of, approach to determine, other things about their case using machine learning, or you want to use something perhaps not as.
[00:13:52] Precise. Or maybe more powerful, but a little bit riskier. So there are possibilities. That transcend, that, like singular, okay, I'm just gonna use this one model that's like this, so I'm a true believer of ensembles. I think ensembles are the way to go with a lot of things.
[00:14:12] And in the book I argue why should you limit yourself to one model, you could use so many of them and figure out what's best over time. You could choose several and just have some kind of voting system and decide based on that.
[00:14:25] The same goes with people as it does with models, often the consensus is far better than the singular person.
[00:14:32] Dr Genevieve Hayes: So now I'd like to turn to the practical mechanics of building interpretable models. Suppose you're starting a new machine learning project and you know that model interpretability is important beyond simply choosing an algorithm that's inherently interpretable, such as a linear regression. What other steps can you take to ensure your model is interpretable?
[00:14:56] Serg Masis: There's a lot I like that very data driven approach. Obviously, when you're discussing a topic. Like machine learning, interability, everything becomes around the model. But I think that's a mistake, which is why I purposely devoted several chapters to things that you probably think, do they belong there?
[00:15:12] But I certainly would argue they do, feature engineering and things like that. You need to apply techniques on the data side. Often to improve outcomes on the model side. For instance if you're interested in bias mitigation because you determine through interpretability techniques that your, model has bias.
[00:15:33] You're not just gonna focus on bias mitigation modeling. You could apply bias medication on the data side. And that's probably even a better way of doing it, tackling at the source, and eventually you get to not just the source of the data, but the source of the bias itself.
[00:15:52] That would be the ideal. But I think there's a lot of things that are missed by taking the model first view, I think the model's a lens through what? See the data, and I think often people see it as the end of the line.
[00:16:06] I throw all the data there, it comes up with a decision. If it's a good accuracy, you just set it and forget it. And to me that's a mistake because there's a lot of missed opportunities. Like you figure you got a relatively good enough accuracy and you may be, start to use all this hyper parameter tuning to get it to the highest amount.
[00:16:26] Maybe you don't realize that by doing the hyper, the parameter tuning , you perhaps biased it in one direction and it caused a systemic flaw in capturing some other nuance, some population of people or something, and you're getting all these false negatives or false positives based on that. And you could perhaps improve this had you dig deeper into these sort of things
[00:16:52] because the model can help you figure out how to improve the model itself. It can help you figure out what's going on with the data. So sometimes modeling is not just the end of the process. It might be even the beginning or the middle. There's a lot of textbooks that actually tell you, okay, you should do.
[00:17:12] Data science in this sequence. And yeah, of course I'm not arguing. You shouldn't do data cleaning and these other things. Of course you can, but sometimes the model can help you do the data cleaning. In other words, you can find data. Problems through a model, you can explore data through a model.
[00:17:30] I think the main element you need to be able to do that is a business understanding. Without business understanding, you can't do any of that. So I think the rest of it is a question of knowing how and when to use a model to improve each part of the process. And so I think that.
[00:17:49] Improving the data through the model is a missed opportunity
[00:17:53] Dr Genevieve Hayes: Can you give us a practical example of that? 'cause that sounds like it could be a very useful technique.
[00:17:59] Serg Masis: Yeah, it is. In agriculture, there's all kinds of time serious problems involving weather. So sometimes I'll have a problem involving a disease and I need to know what the probability that disease will occur in a plant,
[00:18:16] and that has to do with a number of different factors. It has to do with, soil type. It has a lot to do with the weather, the humidity at a certain time, or, what stage the plant might be, because sometimes, depending on the stage, it might be more vulnerable to that disease. But the ones that are more time dependent are the weather ones,
[00:18:36] so I would use a very complex model, an LSTM model and I would figure out what were the optimal windows for these weather variables. And by windows I mean like you have the forecast a few days in the future or if you're looking at historical data, it might be a few days of the future of whatever time step you're looking at.
[00:19:00] When the plant was this big, when the plant was this big, so on, and the plants are crops, and then you might be looking a few days in the past and you wanna see what is the sensitivity, what are the points in which say, these different variables become highly correlated with a certain outcome.
[00:19:19] Like you say what happens if I have a rolling average, let's say, of weather, different time steps, three days in the past five days, six days. What if I use a standard deviation of it, so you have different kind of engineered features. On different timestamps, and you wanna see which one is most correlated.
[00:19:39] So you can just throw them to a an LSTM model, and then you can visualize it using these attribution techniques. And so you essentially have a heat map where you have time on one axis, features on another. So you have, one different row for every feature. Imagine all different colors and you have the bright reds and they're telling you, this seems to be the ones that highly correlate with the outcome. Of course, it's not really a measure of correlation. What it is, it's just attribution. And so I run different kinds of attributions and I get a sense, oh, it seems like the feature I should use is, five day rolling average for humidity.
[00:20:18] A three day in the past rolling average for. Hours of daylight or radiation, that's how I engineer that features because in a lot of these problems, less, it's more, you don't wanna have, a hundred different weather features,
[00:20:33] you wanna have the ones that are best. And so how do you figure that out? There's a number of different ways of doing that, but to me, the most complicated thing has been figuring out the time aspect. Because, some people go by I'm just gonna throw everything at it and then I'm gonna select
[00:20:51] after I trained the model, I trained a few models, and then I do a feature selection there. But you could do a pre-selection based on these attribution techniques?
[00:21:00] Dr Genevieve Hayes: So you're basically using these models as highly sophisticated, correlation based statistical tools.
[00:21:07] Serg Masis: Yeah. Yeah. That's one way of using them.
[00:21:10] Dr Genevieve Hayes: So if our listeners could implement just one technique, this week to improve the interpretability of their machine learning models, what would you recommend?
[00:21:20] Serg Masis: One technique that I love to use and it's a very unsophisticated technique, but to me it's fascinating. Is rule-based models. So a lot of decisions that we take in life or that are taken by companies.
[00:21:35] Whether they're taken by people or even machines, they're not necessarily really complicated. They're more like decision trees, but if we strip them down to their elements, they're not even decision trees. They're decision rules, and you can build models that basically are, you take a decision tree and imagine you go from the root all the way to the leaves and you just extract
[00:21:56] all the decision rolls so say in that tree, you have 16 leaves, you'll end up with 16 rules. So when you have all those rules, you get a sense of what really is going on. That station tree, some will be better than others. And so there's techniques to do that.
[00:22:13] There's a library, I think it's called Eye Models, and I've used it, it has some great, tools for that. And surprisingly, they work very well in a lot of industries. I've been using them in finance for a client. And often when you're making , high stake decisions in finance you want something explainable, but you also want something that's powerful.
[00:22:32] So if you have these kinds of rules, you have a collection of them. You could turn them into a voting classifier. Basically, we're taking several of these rules and, voting between them. This one says I should sell this one says I should buy, and then say you have five, there's gonna be a tie breaker.
[00:22:49] But if they all go in one direction, it's a strong indicator that perhaps you should go one direction or another. So there's some interesting applications to that. I really like decision rules.
[00:22:59] Dr Genevieve Hayes: I've once had a use case in one of my jobs where we're trying to reduce the workload of , a team of staff who were manually reviewing text messages to see if they were worth going on to a more thorough review and. , I fitted a decision tree in order to assist with this.
[00:23:22] And the first rule in the decision tree, which was does the message contain numbers, was so powerful that we were actually able to just use that one rule and reduce their workloads massively. Because basically after we introduce that one rule. They didn't need to review anything that had a number in it.
[00:23:40] We just automatically assumed that it was relevant and it got passed on to the next team. That's how powerful this technique can be.
[00:23:48] Serg Masis: Yeah. Yeah, I agree. It is, it's powerful.
[00:23:52] Dr Genevieve Hayes: So for listeners who wanna get in contact with you, Serg, what can they do?
[00:23:56] Serg Masis: I'm pretty active on LinkedIn. , Sometimes more than others. But , LinkedIn is your best shot, if not, I have a personal website. Serg.ai. You could send me a message through there if you want as well.
[00:24:08] Dr Genevieve Hayes: There you have it. Another value packed episode to help you transform from technical implementer to strategic expert. If you enjoyed this episode, why not make it a double next? Week catch Serg's value boost a 10 minute episode where he shares one powerful tip for creating real strategic impact. Right away.
[00:24:30] Make sure you're subscribed so you don't miss it. And if you found today's episode useful and think others could benefit, please leave us a rating and review on your favorite podcast platform. Thanks for joining me today. Serg.
[00:24:45] Serg Masis: Inviting.
[00:24:45] Dr Genevieve Hayes: And for those in the audience, thanks for listening. I'm Dr. Genevieve Hayes, and this has been Value-Driven Data Science.