Value Driven Data Science

[00:00:00] Dr Genevieve Hayes: Hello, and welcome back to Value-Driven Data Science, where data professionals become strategic experts. I'm Dr. Genevieve Hayes, and I'm here again with Adam DeJans Jr., co-founder of AI and decision intelligence consultancy BitBros, and co-author of The Decision Factory, a novel about decisions under uncertainty.
[00:00:24] Last week, Adam and I discussed how data scientists can shift from modeling data to designing and building the systems or decision factories stakeholders actually use to make decisions. Today, in this Value Boost episode, you'll learn why no single discipline, including machine learning, is enough to build a good decision system and what to do about the gaps in your own skillset.
[00:00:53] Welcome back, Adam.
[00:00:54] Adam DeJans Jr: Thank you for having me back again, Genevieve
[00:00:57] Dr Genevieve Hayes: Data science has effectively become synonymous with machine learning but when it comes to enabling better stakeholder decisions machine learning isn't enough Adam, in your book you define what you refer to as the modern decision stack a collection of disciplines you believe are necessary to enable those better decisions what disciplines make up that stack and what does each one provide?
[00:01:24] Adam DeJans Jr: Yeah, so the eight layers that we introduce in the book are optimization, probability, machine learning, simulation, tuning, operations, LLMs, and the people themselves as well. So the optimization layer is really how do you make optimal decisions, as optimal as they can be, given that, your model's at some fidelity that may or may not represent the world all too well.
[00:01:52] Probability layer is just basically saying, "Don't use point forecast." Actually look at the distributions. So you wanna be able to model risk. You wanna be able to look at uncertainty, the variability within the model. Probability lets you do that. Machine learning lets you discover forecasts and different patterns.
[00:02:11] It's what everybody is used to. Simulation, the people coming from optimization usually are aware of this, but not all data scientists do it. But simulation is really a safe place to practice making decisions before you launch it into the real world, and it costs you tons of money. Tuning is the exercise of like a feedback loop, when you make a decision or even in your simulation, learning different parameters or updating your policy, really honing in on it to make it better.
[00:02:44] Now operations and people are tied together. The operations teams at companies really have the deep domain expertise, they know the real world constraints. They have domain knowledge, and they help you build your model.
[00:03:01] The people generally are your stakeholders. Those are the people you wanna make your recommendations to. Last time we talked about recommendation engines, like letting them still be in control while giving them solutions. And LLMs is a new one. LLMs is now just empowering people to use a lot of these technologies themselves.
[00:03:22] So the best example is what if engine. In natural language, some operations planner wants to ask, "What if there's a tariff next month? What if tariffs go up by 2%? How does that affect things?" And then the stack will be able to run the simulation, run through the probability, do an optimization, look at the decisions that come out and different metrics.
[00:03:46] That's how those all interact together.
[00:03:48] Dr Genevieve Hayes: So what happens if an organization leans on just one layer of that stack for example, machine learning instead of using the entire stack?
[00:03:57] Adam DeJans Jr: Yeah, you're doomed to fail. So I guess not, because a lot of companies don't fail, and they use only one of these. But You're not gonna get the best decision that you can, and so you're not gonna run the best business that you can. You need to mix all of these together in order to have a coherent system that really takes you to the next level.
[00:04:18] If you just do machine learning and you're gonna build a forecast, but what do you do with it? Do you just order to the forecast? No, because it has different consequences whether you're over or under. And so you wanna be able to apply economics to this in order to make a decision, and you wanna see what happens when you do that, and then you wanna be able to make sure this is actually implemented.
[00:04:40] So it all comes together
[00:04:42] Dr Genevieve Hayes: Are there any layers you find organizations are more likely to overlook?
[00:04:47] Adam DeJans Jr: Probability is a big one because it's easy to just wish it away. Even the people that do op- some of them don't have strong optimization layers, but a lot of people use MIPs, and with a MIP it's very easy to just ignore probability because MIP solvers are deterministic naturally. And then when you add uncertainty to them, people think, "Oh, I gotta do stochastic optimization." They think of two-stage, then they think of something that's quite complicated. It takes a lot of compute power to run, and then they just ignore it away.
[00:05:22] So I think probability is often overlooked, but it's necessary to really capture risk that is going into your model
[00:05:30] Dr Genevieve Hayes: Before moving into data science, my background was in statistics, so I think in terms of probability, and I would agree with you. Most organizations overlook probability.
[00:05:41] Adam DeJans Jr: It doesn't need to be precise even. It can be as simple as the chance of this coming out is high, medium, low, very low. Even something like this, you can go quite far. It's gonna be better than your deterministic solution, because it allows you to look at these different scenarios.
[00:05:59] Dr Genevieve Hayes: People focus too much on severity and not enough on likelihood of an event,
[00:06:04] Adam DeJans Jr: That's right
[00:06:05] Dr Genevieve Hayes: Data scientists are typically stronger in machine learning but weaker in optimization and simulation. How would you suggest they go about closing the gaps in their skill set?
[00:06:17] Adam DeJans Jr: First I'm just gonna say that I think the optimization community has a huge problem of trying to make things complicated. And I'm from the community. If you open any optimization textbook, you're gonna get confused if you don't have a formal training in mathematics, because the optimization textbook is typically taught around things that practitioners don't really care about.
[00:06:40] You open an optimization textbook, it starts talking about convergences and proofs of optimality and how the simplex algorithm works, which by the way, doesn't even really work like that in real life. That's the textbook version. The modern solvers don't apply it that way that you're gonna learn in the book
[00:06:58] but these aren't things that really help with making better decisions. It's theoretical. All the optimization books are theoretical. I'm not trying to plug myself, but I wrote a handbook for MIP optimization, which was trying to get rid of any theory that isn't necessary and giving you just enough to where it helps your intuition when you go to model.
[00:07:19] So if you wanna get good at optimization, I think in practice, you really need to learn the modeling techniques and not the underlying mathematics of the solver, which is what a lot of books do. Then how do you model better? You have to understand the physics of your problem better. So modeling is quite easy now with LLMs.
[00:07:38] You can get them to go 90% of the way. But what is it that you're trying to model? 'Cause machine learning, you can just download a dataset and then, there's some like SK learn or PyTorch or something, and you can come up with a model, 20 minutes later. Where optimization is not that easy.
[00:07:54] You have to actually understand the physics of your model and then implement that you have to understand what's the objective function, what's the decisions that I can make, what's the constraints, what's stopping me from making certain decisions.
[00:08:09] And because of that, it's hard to just pick up. But yeah, modeling is the key
[00:08:14] Dr Genevieve Hayes: By the physics of the model, is that how an action impacts the financial bottom line of a business, for example?
[00:08:21] Adam DeJans Jr: The physics of the model, things like what is even feasible in your model? For example, in automotive manufacturing. I always use automotive, 'cause I'm from Detroit.
[00:08:33] That's all we have there. But in automotive you have things like the plants can only make so many vehicles per day, you can't make 10,000 vehicles a day. It's not possible. When you talk about when the car gets off the line, and then people need to put some post-production parts on it you have labor.
[00:08:54] That's a constraint, you have to understand how to take constraints that are out in the real world and model them in a way, especially if you're doing like m- modeling, in a way where it's represented so that a solver can actually utilize it. In that case, usually it needs to be linear.
[00:09:11] But yeah, understanding the business and what makes things legal decisions, and then also as you pointed out, what's the economics of this, if there's holding costs because I put an item in a warehouse,
[00:09:24] that cost needs to be reasonable, otherwise when I run my optimizer, that holding cost could override the rest of the model, maybe it's so expensive to hold things, I don't hold anything, and that doesn't make sense.
[00:09:35] Dr Genevieve Hayes: If a data scientist listening to this only has time to focus on strengthening one layer in the coming months, which would give them the best return on their investment?
[00:09:45] Adam DeJans Jr: I would go with optimization, which is counter to what we just said, where probability is one of the more important ones. And I say that because I think having even deterministic optimization ability is something that not a lot of data scientists have.
[00:10:01] So I think just starting to learn optimization techniques and thinking in this type of manner where it's not like a train test split and you're looking at accuracy and things like this, but understanding how do I model something that has like combinatorial number of ways that you might choose and, how do I go about modeling this and how do I represent this faithfully in the real world?
[00:10:25] I think that is an easier step because that's the next natural step, and then after that it's, okay, how do I add uncertainty to this? And then it becomes, okay, how do I simulate this uncertainty? Which usually there's some Monte Carlo simulation, or maybe it's a discrete event simulator, but that's the next natural step.
[00:10:46] And then getting that to production requires the people piece, which is also important. But from a technical layer, optimization is the next easy step for most data scientists outside of the people piece. People piece is always the most important
[00:11:03] Dr Genevieve Hayes: And that's it for today's conversation with Adam. If you haven't already, listen to our previous episode where we discussed why good models don't guarantee good decisions and how data scientists can start building decision factories inside their organization instead. Thanks for joining me again, Adam
[00:11:24] Adam DeJans Jr: Thank you for having me
[00:11:25] Dr Genevieve Hayes: And for those in the audience, thanks for listening.
[00:11:28] I'm Dr. Genevieve Hayes, and this has been Value-Driven Data Science.