Episode 132: [Value Boost] Making a Causal Argument Without Experiments
Download MP3[00:00:00] Dr Genevieve Hayes: Hello, and welcome back to Value-Driven Data Science where data professionals become strategic experts. I'm Dr. Genevieve Hayes, and I'm here again with Dr. Quentin Gallea, author of The Causal Mindset Handbook. Last week, Quentin and I discussed how causal AI combines causal inference and predictive AI to enable better decision-making than either of its components can alone.
[00:00:28] Today, in this Value Boost episode, we'll explore how to build a credible causal argument in business settings where true experiments aren't an option. Welcome back, Quentin.
[00:00:41] Dr Quentin Gallea: Pleasure to be here again and share again my passion, expertise, and obsession about causality
[00:00:47] Dr Genevieve Hayes: When it comes to establishing causality, true experiments or randomized control trials are considered the gold standard, which is why they're standard practice in medical testing. However, randomized control trials can also be very expensive and slow, making them unsuitable for supporting decision-making in most business environments.
[00:01:10] But that doesn't mean that data scientists working in such environments need to abandon causal thinking entirely. In fact, that's exactly when building a causal mindset can help the most. Quentin, what is it about randomized control trials that makes them so powerful for establishing causation?
[00:01:31] Dr Quentin Gallea: So the thing is, if you want to infer causality, you would like to know what will happen without a decision, without a treatment, without a marketing campaign, without a drug, without a vaccine, without a policy. And with randomized control trial, or in AB testing in business, we really build kind of parallel worlds.
[00:01:50] For example, on Netflix, you can observe different users randomly selected with different layout of the front page. And then you can decide, okay, what works better, what increase retention, because we have basically small parallel worlds. These are randomly selected, allowing to have two groups that have no systematic differences except for the treatment.
[00:02:12] And this allows basically, if it's done properly, to infer causality. A small comment I want to do here is that first, causal inference also allows to measure reliably and infer causal effect with observational data. There are plenty of methods that are called quasi-experimental methods.
[00:02:31] And the second thing is that often we also have limitations in randomized control trial. I also teach in medical sector, and for example, we often have limitations with respect to external validity. You will work with inbred strains of rats that are very specific in the lab, but then it's difficult to extrapolate to humans, while we care about humans.
[00:02:52] Or the experiment is in a very short amount of time. It's done one day, one week, but actually the effect might change over time. So again, you have sometimes extrapolation limitation with the capacity to extrapolate external validity
[00:03:06] Dr Genevieve Hayes: So what specifically makes these randomized controls trials so hard to run in a business environment?
[00:03:14] Dr Quentin Gallea: So you can use this widely, but it takes time to set up. It requires usually also certain amounts of observations. And for many business that are not Spotify or Amazon, they don't have thousands, millions of users every day where they can easily test their new features.
[00:03:33] In other cases, for example, in top tech companies, they run hundreds of experiments at the same time every day. So in some companies it's not such a bottleneck, but for most companies it is due particular to technical knowledge and to data availability. So as I like to frame, most questions are causal and most answers are not.
[00:03:54] Why? Because most questions are causal and we would love to have a clean experiment or clean quasi experimental methods to measure the effect, but it takes time, it requires a lot of data, And for the vast majority of decision actually, we don't have this in business or beyond
[00:04:10] Dr Genevieve Hayes: So we're not Google, for example, and we can't test what happens if you change this button from red to blue .
[00:04:17] Dr Quentin Gallea: Exactly
[00:04:18] Dr Genevieve Hayes: In your book, you outline a framework to help data scientists and business leaders apply causal thinking when randomized control trials aren't possible.
[00:04:27] Could you quickly walk us through that framework?
[00:04:31] Dr Quentin Gallea: For all these other decisions that we have every day even in our personal life or questioning politics, reading the news, all these causal claims, we basically need a different toolkit. And that's really where it started with my book and with my method is instead of discarding entirely the science of causal inference, let's just use the concepts to know where to look and how to challenge these claims.
[00:04:55] So the causal mindset is a combination of two components. The first, it relies extensively on causal inference concepts. It teach, okay, when you have a causal claim, what is the reference group? What is used as the counterfactual? Then it allows to ask the right question. For example, is there something else affecting the effect that I observe?
[00:05:18] Here you look for what we call confounders, common cause that will bias the estimation. And then there are different steps like this, but these are the most important ones. The second thing I do in my technical practice and also non-technical advisory is I combine causal inference in this causal mindset with system thinking.
[00:05:38] Because I've realized that in practice, you can have perfectly causal estimation and still make a mistake and take the wrong decision. Why? Because people will over interpret the results, or basically, I like to refer to statistics here, to type three error, because many people know type one and type two in statistical testing.
[00:05:58] Type three is answering correctly the wrong question. In business, it happens. You answer correctly, but you looked at the wrong metric, or typically, when you... One of my expertise today is measuring the impact of generative AI, and what people do is they answer the wrong question. They look at the short-term, easily available metric, and this tends to change over time or hides negative longer term effects.
[00:06:22] So you look at a boost in productivity on the short term, but actually this boost in productivity in your marketing team because they can create a lot of AI slop and more content, you have more views, but it affect brand loyalty which eventually is negative for your growth and brand perception, which is also negative for long-term growth.
[00:06:41] So what I do is also I combine this causal mindset with system thinking, where you really try to ask, "What's the good question I have to ask? What are the metric?" And usually this is multidimensional. It's not just one. So I teach people to think causally, but also to always look for the other side of the coin, to play the devil's advocate.
[00:07:01] Every time someone serves you a dashboard or a metric, you always ask, "Okay, even if this is going in the right direction, how could it backfire? What could be the blind spots I have?" And so you can fill this blind spot by checking for other metrics, asking the right questions
[00:07:17] Dr Genevieve Hayes: Oh so it's not just about creating a substitute when you can't use randomized control trials. S- initially I was thinking maybe this would be some sort of quasi-experimental solution, but this is when you can't even apply quasi-experimental solutions.
[00:07:33] Dr Quentin Gallea: Exactly. It's really non-technical. It's really a mindset. So indeed, quasi-experimental methods are absolutely valid methods in causal inference. They might be actually at least as time-consuming as randomization and randomized experiments, which requires a lot of expertise, a lot of data and many things, and time to analysis.
[00:07:52] But actually, the mindset is really a mindset. It's really something I can teach to top executive, to C-suite, to leadership teams, to boards, to ask the right question, challenge the dashboards and make better decisions. Basically, you present something to an executive.
[00:08:07] You show, okay, our new gen AI or chatbot increased productivity, and the executive, either they buy it directly because they want confirmation bias, or they want to challenge it a bit, but they don't know how to. And basically, what I teach is, here is where you should look. Okay, you can ask first, what's the reference point?
[00:08:27] When you say productivity increased with our chatbot, what's your reference group, ? Oh, it's a pre/post evolution. You look before, then you release, and that productivity goes up. So that's the first question. Once you have this in mind, I teach the second step. Okay, now that you understand the reference point, you can ask, "Is there something else?"
[00:08:46] It's simple question. Is there something else that might explain this effect? Something else than the treatment here, the chatbot release. What about seasonality? We know that eventually seasonality, , affects productivity. So when we release this chatbot in January, how is productivity. Is it usually higher or lower? Is there no change? So you can use domain knowledge to at least start challenging the claims and see, okay, if usually you're more productive in the periods where you released , the new chatbot even if you're not technical, you can say now we are not really certain,"
[00:09:20] we are not sure what is happening. Is it the chatbot? Is it seasonality? Is it both? You won't have the answer with this qualitative analysis, but it helps to evaluate the uncertainty, and it empowers people against all these claims, even non-technical people
[00:09:36] Dr Genevieve Hayes: And it's helping people not to fall into the trap of mistaking correlation for causation
[00:09:42] Dr Quentin Gallea: small comment when I say this because every time I talk about this, people say, "But we all know correlation does not imply causation." Let me tell you something. They don't. And I talk extensively to many people. They can repeat the mantra, but actually once they are served causal evidence, they forget about it, and they don't know where to look.
[00:09:58] And I share on LinkedIn every now and then, top papers, for example, in Nature, in Science, in top scientific journals where they make the mistake between the title and the abstract. You don't even need to open the data to see that they interpreted causally something that is correlational, so definitely there's a lot of space to improve our decision-making and interpretation of data
[00:10:19] Dr Genevieve Hayes: I have a PhD in statistics. I literally have taught students correlation does not equal causation. And it took me way too long for the penny to drop that, oh, and machine learning models are correlation-based models. That was the bit , that no one ever connected for me during my education, and it took way too long for me to connect myself
[00:10:43] Dr Quentin Gallea: And that's exactly why usually in my talks, workshops, I say, "Okay, we all know correlation doesn't imply causation, but why?" And you need to explain why it's not the case such that you can disentangle and separate when you can infer causation and when you can't. And back to machine learning, I spend a lot of time explaining the terminology because, for example, if you have a strong predictor, you can totally say, "Okay, I observe that people in business who use extensively AI have promotion faster." Here, there are no causal claims. It's just you observe people who use AI.
[00:11:12] You observe that they are promoted faster. Almost 100% of people who will hear this claim will infer causality. Even if here in my wording, I'm not saying one causes the other, I'm just saying I observe the two. And so that's why I also do a lot of communication on how to explain and report these results because you can very easily slip from prediction to causation and the general public will misunderstand what you mean or even technical people.
[00:11:36] So the words are not a technical aspect, but it's very important
[00:11:39] Dr Genevieve Hayes: You've taught thousands of professionals from data scientists to executives over the course of your career. If you could get those people to understand just one thing about causal reasoning, what would it be?
[00:11:51] Dr Quentin Gallea: Yeah, I always pause during my workshop talks and I just say exactly this thing. " if there is one thing you have to remember about me, if there's one thing that I'm doing all the time, any time I hear because or therefore, I hear, I read, or I say it," apply it to yourself as well.
[00:12:09] Any time you hear or say because or therefore, you just pause, because here you're making a causal assumption, and you ask, "Is there something else? Is there something else that might affect this relationship or explain it entirely?" And here, basically what you're looking is slightly more complex than just something else.
[00:12:27] It's what we call a confounder. You need to think about something that affects at the same time the outcome and the treatment. So if you observe Again back to the example from the first episode. You have marketing that increase and sales increase at the same time. Is there something else?
[00:12:43] Probably seasonality. We spend more during high-demand periods, so it's confounded with this effect. You could think about how it might be the other way around. I observe that when we spend more in marketing, we have actually lower sales. Maybe you signed a turnaround budget, you started spending more other regions that were weaker, where your people who'll buy less of your product.
[00:13:08] So here it could be also the other way around. You observe a negative relationship, but this might be driven by the fact that you spend where people consume less. And so just pausing and asking, "Is there something else?" And this is tremendously powerful for challenging political claims and everything in the news.
[00:13:28] The biggest red flag I see online is research has shown. Because, they use this as a notoriety argument, and most people don't read research and don't un-understand it, or both. So it leads to a lot of misinformation where they say, "Okay, MIT researchers show this." No. And I teach, okay, just pause, look at the results, usually associative, and you ask, "Is there something else?"
[00:13:51] And again I see top influencers in health, for example, in medical research, really some of the biggest voices in an influencer in this realm. And most of their evidence is correlational, and they make the jump to causal claims. And you can just challenge this with this simple non-technical tool.
[00:14:07] Is there something else?
[00:14:09] Dr Genevieve Hayes: And that's it for today's conversation with Quentin. If you haven't already, listen to our previous episode where Quentin and I discussed how data scientists can develop a causal mindset in a predictive world and enable better executive decisions. Thanks for joining me again, Quentin
[00:14:28] Dr Quentin Gallea: Thank you very much for the invitation
[00:14:30] Dr Genevieve Hayes: And for those in the audience, thanks for listening.
[00:14:33] I'm Dr. Genevieve Hayes, and this has been Value-Driven Data Science.
Creators and Guests