Value Driven Data Science

[00:00:00] Dr Genevieve Hayes: Hello, and welcome to Value Driven Data Science, where data professionals become strategic experts. I'm Dr. Genevieve Hayes, and today I'm joined by Adam DeJans Jr. Adam is a decision scientist and optimization expert who has led high-stakes decision systems in complex, uncertain environments at companies including Amazon, Toyota, and Ford.
[00:00:28] He is the co-founder of AI and decision intelligence consultancy BitBros and co-author of The Decision Factory, a novel about decisions under uncertainty. In this episode, we'll explore how data scientists can shift from modeling data to designing and building the systems or decision factories stakeholders actually use to make decisions.
[00:00:54] Adam, welcome to the show.
[00:00:56] Adam DeJans Jr: Hello. Thank you for having me on. Very excited
[00:01:00] Dr Genevieve Hayes: Data scientists are trained to model data, and yet it's rare that a stakeholder wakes up hoping that a new predictive model will be waiting in their inbox when they arrive at work. What most stakeholders really want is help making better decisions, which is exactly where traditional data science falls short and the adjacent field of decision science excels.
[00:01:24] But what does it actually look like in practice when data professionals prioritize enabling better organizational decisions over making better predictions? And how does such an organization differ from the proverbial data-driven organization so many companies claim to be striving to become?
[00:01:43] This is the exact question, Adam, that you and your co-author set out to answer in your excellent book, "The Decision Factory," which is essentially a 200-page worked example of what it takes to build and deploy what you call a decision factory. To begin with, what exactly is a decision factory, and how does it differ from the machine learning pipelines most data scientists are familiar with building?
[00:02:11] Adam DeJans Jr: Yeah we kinda... I don't know if this was a term, but we coined the term, if it wasn't already, a decision factory. A decision factory is basically an operating system for repeatedly making, evaluating, and improving your decisions under uncertainty, and we claim that any decision you make is almost always under uncertainty.
[00:02:32] You rarely have perfect info- when you're talking about machine learning and the machine learning pipelines, the end output of that is typically some prediction model, which is, let's just say a forecast. It could be other things, but typically it's some type of forecast or dashboard.
[00:02:48] But that's not the end of it. That's really just the beginning, because that forecast typically gets used to make some decision. So I think the big difference is that When we talk about decision factories, we're talking about the end goal, which is to make decisions. And you're not just doing this once.
[00:03:05] Something that happens again and again, you end up doing it sequentially, and you do this in a way where you want to update based on new information that arrives. In our book, We talk about planning versus policies, which I think is important here. When you think about a plan, it's something you set and you expect it to play out, where policy is really how you make the decision.
[00:03:28] It's not really the decision that you make, but it's the mechanism for how you make the decision. So that could mean many different things. If you look at Warren Powell's frameworks, he classifies these into four decision policy classes. So that's what we mean when we talk about a decision factory, and I think that's what would separate this from the traditional ML pipeline, is thinking all the way through end to end.
[00:03:54] And in fact, I might even say the ML pipeline is a part of this factory as well. It is a part of it
[00:04:01] Dr Genevieve Hayes: I saw them as operating at two different levels of abstraction. So The ML pipeline worked as a component of the decision factory, but you needed the decision factory layer over the top to connect its outputs back to the business needs. Is that how you see it?
[00:04:18] Adam DeJans Jr: Yeah, that's exactly how I see it. 'Cause at the end of the day what are people trying to do? Every business is essentially trying to make more money, or if it's a hospital, maybe they're trying to be more efficient. Even they're making money in the US sometimes, so different parts of the world.
[00:04:34] But yeah, money's a big motivating factor. As Paul says all the time, and I take it from him, is if you want to run a better anything, you need to make better decisions. And almost everybody really agrees with that. If you wanna make more money, you need to make better decisions. If you wanna run a more efficient process that comes down to making better decisions.
[00:04:53] And forecasts and a lot of these ML pipelines and ML models, they're feeding information to you to help you make the decision. So that's why I say the machine learning pipeline typically has an end output. So if you think about a forecasting model, your goal really is just to build the forecast, and then somebody else will use it downstream,
[00:05:15] and basically all these machine learning models end up just creating data that gets used for decision-making. So it's just one piece of the pipeline. It's getting more information. And that's what typically people are using ML for. You're not gonna make something just for fun, although you see this happen.
[00:05:34] Sometimes you see this happen with a lot of data scientists who are very interested in the technical side of things, but then they forget that this is really being used to make a decision. But usually that's done by a business person. They're not involved in the process, but it very much gets used.
[00:05:52] So I think they're at two different stages, but it's all one big machine because everything comes back to the decision. 'Cause in order to run a better anything, you need to make better decisions
[00:06:02] Dr Genevieve Hayes: In your book, the company at the center of the story, Fulcrum Logistics, has a data scientist, Liam who's responsible for building machine learning forecasting models. Liam's competence is never in question, and yet at the start of the book, the company's decision systems are a complete train wreck.
[00:06:22] Why isn't building good models good enough to guarantee a good decision system?
[00:06:29] Adam DeJans Jr: Yeah, that's a good question, and You see this a lot of times from the upper level management and executives there's always an initiative, "We need a better forecast. We need a more accurate forecast." But the goal isn't to make a better forecast, the goal is to make a better decision.
[00:06:44] If for some reason executives could take two rocks and bang them together three times and spin around, and it gave them a better decision, that's what they would do. They don't care how they get it, they just wanna make more money. But with the forecast the thing about them is the future isn't a number.
[00:07:02] So the organizations routinely, collapse uncertainty into one number. So you might hear something like a demand forecast of 100 units has a lead time of 14 days and has a price of $50. And then a lot of people take that as fact when they go to make their decision, or fact when they put it into their optimization model.
[00:07:24] But really we need to be treating these as probabilities, they might say demand is forecasted at 100 units. That's never gonna happen. It might be 60, it might be 140, it might go all the way up to 300, who knows? But we can put probabilities on each of these and use these when we make decisions,
[00:07:42] 'cause that 100 might be very like the real value that comes might be really close, centered around that 100. It might be a really tight variance there, or it could be really wide, and that's something that decision makers really need to understand.
[00:07:55] Dr Genevieve Hayes: That was what I really liked about your book. The models that you describe don't just have point outputs. You gave a point estimate, an uncertainty range, and a tail risk, or did you call it conditional value at risk?
[00:08:10] Adam DeJans Jr: Yeah, that's what people typically use, I think we called it something else in the book, but yeah, I think I don't remember the exact term. We tried to make it so that anybody could read the book and understand, but yes, same concept
[00:08:23] Dr Genevieve Hayes: My background's actuarial, so I just kept reading tail value at risk 'cause that's
[00:08:27] what we call it. But yeah. And I also loved the fact that you raised the idea of using machine learning forecasts as random variables rather than point estimates. This was an approach that I used in my own PhD research, but I've rarely seen people actually use it in the literature. So can you explain what you're actually doing when you treat a forecast as a random variable, and what impact does that have on system design?
[00:08:57] Adam DeJans Jr: Yeah, so when you treat a forecast as a random variable, which goes back to the last answer you have uncertainty in your prediction. The decision problem becomes much richer now. You can now ask things that you couldn't ask before. Like what happens if I'm wrong in my forecast?
[00:09:15] It's gonna be in one direction or another with some type of probability. I can look at things like what's the upside? What's my best case? What's the downside? What's my worst case? Maybe I look at the P90 or the P10. Everything I believe when you make a decision comes down to economics.
[00:09:32] So a lot of the optimization people, they're really doing applied economics. And with probabilistic forecasts in particular, but you can apply this beyond forecast. But, now you can really look at like what's the cost of stock out versus overstock? And I know that these typically hurt me in different magnitudes,
[00:09:54] like I look at the car dealership example where if we overstock on cars, then what happens? You pay a little bit more insurance. It sits on the lot a little bit longer. It's doesn't really cost you that much. Now what if you under stock and you stock out? Typically, I don't think the customer might wait around.
[00:10:11] They might go to a different brand. You gotta look at the lifetime value of these customers. And you can't really do this if you don't really understand the full array of outcomes. So when you reduce everything down to a single point you lose quantifying what's the chance of me going overstock and under stock?
[00:10:31] By how much in either direction might I go? And you start to lose this when you bring back everything down to a point
[00:10:39] Dr Genevieve Hayes: Sam Savage refers to it as the floor of averages which he summarizes that if you use averages then you're going to be wrong on average or something to that effect. He expressed it a lot better than me
[00:10:51] Adam DeJans Jr: Yeah, I think the main thing is uncertainty isn't something that you can remove before you make a decision. You can act like it doesn't exist, but it does exist. And my claim is that uncertainty should be an input into your decision-making process, 'cause you can bound the uncertainty.
[00:11:08] If you have probabilistic forecast, oh I might have a very good idea of the range it might fall in, which if I'm selling different products, it might be very tight or it might be very wide, and depending on if it's a tight or wide range, I might make a different decision.
[00:11:23] But typically people just throw that away and they so focused on accuracy and the forecast model, but that doesn't really matter. What matters is are you making good decisions?
[00:11:35] Dr Genevieve Hayes: So what I'm hearing here is when we're dealing with decision-making under uncertainty, which is basically all decision-making because the world is uncertain, We can't just use point estimates. We also have to allow for things like uncertainty ranges and tail risk, and one way of doing that is by treating machine learning forecasts as random variables rather than point estimates.
[00:12:01] Another key element that I saw in your book was the notion of simulated learning. How does simulated learning come into the mix?
[00:12:11] Adam DeJans Jr: Yeah, I just wanna say one thing before I answer that is in the preface of the book, it says, "This is a book about making decisions when the future is uncertain," which is to say this is a book about making decisions. So that's like my favorite line, and it's the first line. So with that
[00:12:30] here's the thing about simulation. Simulation is a very valuable tool because you wanna be able to test things before it goes out live. So you wanna know if some model or some decision-making process or some policy, as we call it, is a good idea or not a good idea. You can typically just test that without risking a lot of the downside of it,
[00:12:53] I don't wanna deploy something that's a terrible idea and then I lose a bunch of customers, I lose a bunch of money. I don't wanna do a lot of learning while doing. I wanna be able to test this ahead of time, which the simulation allows for. Of course, there's things that you just cannot simulate,
[00:13:11] like, when I worked at Toyota you can't simulate the whole of Toyota, there's certain aspects of it you can, but on the whole, there's just some things you just cannot do 'cause you don't know how the competitors will react. You don't know how customers change over time.
[00:13:26] You don't know about the tariff situation. You can model these pieces individually. You can hack something up that works on the whole. But you'll never get it right. You'll never model down to the atom, and you don't really wanna do that either. But sometimes you do need to let things out into the wild and see how it does, and you need a process in order to make a better decision next time.
[00:13:48] So you need this feedback loop. But the simulator allows you to do this offline. Now, simulators are typically hard to build as well, so they can take a long time to build if you want good fidelity. You want it as accurately as you can represent the real world, but you also don't want it to take, 10 years to do a run,
[00:14:06] so there's a balance between fidelity and computation there. And the simulator just allows this feedback loop to happen offline so You can prepare your policies to behave well when they reach the real world
[00:14:20] Dr Genevieve Hayes: Okay. What do you do when you can't use a simulator? You gave all those examples the tariff situation your competitors, et cetera. How do you know when you've got a situation where a simulator won't work, and what do you do when it won't?
[00:14:34] Adam DeJans Jr: That's a good question. I don't know that there's a good way to know that you don't know that it's gonna work. I always tend to try, I'm over-reliant on simulators 'cause they've worked so well for me in the past. Some of those cases that we did talk about is like tariffs,
[00:14:48] You can come up with models, you don't know how good they're gonna be. But you can simulate something. Now, you can't simulate the whole end-to-end supply chain, or maybe there's some new product you're launching and you just can't simulate it, you have no data on it, you haven't sold something similar.
[00:15:05] You think you have something maybe through surveys or some type of data collection, but you're not really sure. But the important thing is when you do this, you need to have some feedback loop that you can react to, you don't wanna keep making a bad decision over and over again.
[00:15:21] So that's why when we talk about policies as a way of making a decision rather than the decision itself we can tune those over time, and that could be something that you do live. You could, set the price of a product, let it run, measure it learn from it, repeat with that information, maybe set a new price.
[00:15:42] Those are hard to do. They're expensive to do because they're in the field, and that's the reason why we try to do simulation as well. But sometimes you just can't. There are certain techniques I'm not the best at this, what we call learning while doing, 'cause I typically have simulators for what I need.
[00:15:58] But I know this is a very open-ended research topic. You'll hear Warren Powell talk about this needs to be a focus of PhDs, in this upcoming wave, is like learning while doing because it's so complicated. But there's things like the knowledge gradient and there's different methodologies I'm not well-versed in them.
[00:16:18] I know they exist. I don't know how good they actually work in practice, but there's people thinking about this
[00:16:25] Dr Genevieve Hayes: By learning by doing, are you talking about experimental learning there?
[00:16:29] Adam DeJans Jr: Essentially, yeah. So if you can't simulate, you're experimentally learning. But you wanna do that in a way that it's the same as if you were to do simulation, it's just more costly. You still have this feedback loop, it's just now, it costs you real money because you're deploying it out into the real world.
[00:16:47] So people tend to do safe things when they do this, and maybe small incremental changes so they don't damage the business overnight, where if you had a simulator, you could test whatever you want, the wildest dreams
[00:16:59] Dr Genevieve Hayes: Suppose one of our listeners builds their own decision factory, how do they take its outputs and actually change how stakeholders make decisions?
[00:17:08] Adam DeJans Jr: Yeah, this is probably the most important question because even if the technical team nails everything and they can't communicate this to the business stakeholders, nothing's really gonna change. The one thing that I always talk about is meeting people where they're at.
[00:17:23] Especially with terminology and jargon, you don't wanna go to your business stakeholder and start talking about myth gaps and talking about P values and all this stuff. They don't really care about it. They don't really understand it. If they did understand it, you wouldn't be there. What's the point of having you if they could do all this?
[00:17:40] Part of your job is to be that translation for them and to help them make better decisions and understand what are they trying to improve So I think really working with your stakeholders, understanding what is their pain points, and then how you will address them. And then the other big thing is not forcing your solution onto them.
[00:18:02] So the MIP community loves to say everything is optimal, oh you're talking about Gurobi it's like optimal solution or FICO, same thing. All the MIP solvers. Very proud of the optimal solution. But I claim there's really no such thing as optimal solution in reality, because there's uncertainty involved, and by the time it gets implemented in practice, it doesn't look like the same model that you ran your solver on anyway.
[00:18:27] And so things that I found to work very well is talking about different trade-offs that might happen. It's not just like one optimal solution, really. You might look at a Pareto front on different metrics that the business cares about, maybe somebody's really into the profit okay, we wanna make a decision to really increase the profit, but they don't really wanna go all profit.
[00:18:48] They want market share as well. So what's the balance between profit and market share? Then you could even show them this Pareto curve and say, "Hey, where do you wanna be on this? Do you wanna be all marketing, all profit? Of course, you don't. You really wanna be somewhere in the middle. Where are you comfortable being?"
[00:19:05] It's like giving them options because , the business also has intuition that you don't have for this from working on the business side. There's things you didn't account for in the model. There's things you never can really account for in the model. Some of it really is just like gut intuition that they have.
[00:19:22] That they can't express in a way that you would be able to put into a model. So I like to treat it all as recommendation engines. I don't ever force anything onto them, especially in the beginning. Eventually, you wanna automate everything and have it auditable and verifiable.
[00:19:38] But in the beginning, it's especially true that you need to really partner up with your stakeholders and give them choices.
[00:19:48] Dr Genevieve Hayes: It's an interesting term you've used there, recommendation engine, 'cause it brings to mind things like the Amazon recommendation engine or the Netflix recommendation engine. Netflix or Amazon doesn't say to you, "We have determined that the optimal movie for you to watch next is whatever," because that would be stupid.
[00:20:08] They just say, "Here are five movies that we think you'd enjoy," and you can choose to watch them if you want, or you can go and choose something of your own. It sounds like that's what you're essentially proposing doing to your stakeholders, saying, "Here are the decisions that we recommend that you make given these different trade-offs.
[00:20:29] You can make them if you want, or you might choose to override them." Does that sound right?
[00:20:35] Adam DeJans Jr: Yeah, that's exactly right. And I've built very simple systems that stakeholders love them because they still felt in control of them. And then you can gain their trust from this, and then you can move to the next level.
[00:20:48] No-no-nobody who's been planning doing things in Excel for the last 15 years, and then you come along like, "Oh, I have this model it's the best answer. Oh, it's mathematically optimal. There's no better answer." And then they're like, "Yeah, you're full of it. It's not the best answer 'cause this reason and that reason," and they come up with a lot of operational things you never really thought about.
[00:21:10] But if you start simple, it doesn't need to be optimization. It could be rule-based. It could just be like, if this, then that. Very simple things, just even starting with automation and then building their trust over time is super important, and I think it's something that's lacking from a lot of practitioners.
[00:21:27] So I almost always treat it as a recommendation system until the stakeholder's finally no longer really checking it, they trust it, and now I can begin to automate, and then I'll have guardrails, of course. If something seems like it needs a human touch, it'll get flagged.
[00:21:47] But yeah, starting simple I like to say complexity needs to earn its spot. And this is especially true in supply chain, where people have been doing it the same way forever, if you go to a tech... Like Amazon, it's gonna be a lot different those people they're in tech it's founded on tech.
[00:22:05] That's their main thing. But you go to a Toyota, it's 100-year-old company. They come from manufacturing. They're used to Kaizen, which is small incremental improvements, and a lot of manufacturing is like this. So it's a slow process, but it's a effective process
[00:22:23] Dr Genevieve Hayes: If a data scientist wanted to build one small piece of a decision factory into their current work without a full organizational overhaul where would you suggest they begin?
[00:22:35] Adam DeJans Jr: I would start with the people and understanding what it is that I'm actually trying to build. And I think this is almost universally true, so it's a cop-out to your answer in a way. But you need to understand, what is the decisions that are going to be made?
[00:22:50] So a lot of data scientists don't know that necessarily. They think they might know this, but They might have a different version of what is actually being decided. And I think if they understand what decisions are even possible rather than what am I being asked for.
[00:23:05] Typically you're being asked for something that isn't directly related to the decision. So the forecast is a really easy one, because you're asked to build a forecast, but they don't actually want the forecast, they wanna make a better decision. What goes into that decision? And that's why the book really emphasizes the three questions, which is what decision am I making?
[00:23:27] And what's the sources of uncertainty? And listing those out will get you a long way. So I think as far as the decision factory is concerned, understanding what are you actually building for is the most important piece. Actually people is listed in the book as a layer of the stack, because of this, because I worked so long with so many different companies and it's always the same thing.
[00:23:54] There's a disconnect on what are we actually trying to do and how do we do it better
[00:23:59] Dr Genevieve Hayes: For listeners who want to get in contact with you, Adam, what can they do?
[00:24:03] Adam DeJans Jr: Yeah, so I have LinkedIn of course, and you can always add and message me there. But I also run a consultancy with John Elum. It's very low volume at the moment. We're very selective due to time that we have. But I think if you go to bitbrosdata.com you'll find more information about that.
[00:24:24] Or if you go to my LinkedIn, which is /addejans. If you just search my name, you'll find me there. You can send me a message there as well
[00:24:35] Dr Genevieve Hayes: And that's it for today's episode of Value-Driven Data Science. But if you want more from Adam, next week you can catch our Value Boost episode where you'll learn the eight disciplines that comprise Adam's modern decision stack and why leaning on just one leaves your entire decision system exposed. And if you found today's episode useful and think others could benefit, please leave us a rating and review on your favorite podcast platform.
[00:25:04] That way, we'll be able to reach more data scientists just like you. Thanks for joining us today, Adam.
[00:25:11] Adam DeJans Jr: Thank you very much for having me
[00:25:13] Dr Genevieve Hayes: And for those in the audience, thanks for listening. I'm Dr. Genevieve Hayes, and this has been Value-Driven Data Science.