Episode 38 – The Art and Science of Survey Design
Download MP3[00:00:00] Dr Genevieve Hayes: Hello and welcome to Value Driven Data Science brought to you by Genevieve Hayes Consulting. I'm Dr. Genevieve Hayes and today I'm joined by Kyle Bloch to discuss the art and science of survey design. Kyle is Head of Research at Gradient. An analytics agency that combines advanced statistical and machine learning techniques to answer difficult marketing challenges.
[00:00:24] He holds a master's in spatial analysis from the University of Pennsylvania and has spent his career helping managers use data to make important decisions. Kyle, welcome to the show.
[00:00:37] Kyle Block: Thank you, Genevieve. Very, very grateful that you're having me.
[00:00:40] Dr Genevieve Hayes: From Buzzfeed quizzes to the national census, it's impossible to get through life without encountering surveys. As a data scientist, I think this is great because it means more data for me to work with. However, not all surveys are created equal. As with everything else in data science, garbage questions going in will inevitably lead to garbage answers coming out.
[00:01:06] So today we're going to take a look at practical techniques you can use in designing surveys to ensure the answers they produce are valuable, as well as different approaches you can take to analyzing survey results to realize and maximize that value. Now, the emphasis here is very much on the practical side of survey analysis.
[00:01:30] And this might seem surprising to some of our listeners given your job title, Kyle, as Gradients Head of Research. Now I'm not sure what our listeners are thinking, but I'm originally from an academic background. And when I first saw your title was Head of Research, the thing that sprung to my mind was that you were immersed in the theoretical side of survey design, far away from the practical side of things.
[00:01:58] But when I started to look at your LinkedIn profile and what you've been doing, I realized that's not actually what your role involves, is it?
[00:02:08] Kyle Block: No, it's quite practical. It's extremely practical. And that's what makes it fun and fulfilling. And there is a, and this might be a surprise to some listeners. There is in fact, a theoretical methodological Academic piece to the world of survey research, and that might be surprising when most of your familiarity with surveys come from what you do see on BuzzFeed or some poorly worded email poll, but there is a whole theoretical world to inform what good survey research, survey design, sample design and interpretation should follow.
[00:02:50] And the beauty of what we get to do with Gradient is apply that in very, very practical realities to help organizations of different sizes, of different sectors, make some really important decisions with generalizable certainty.
[00:03:08] Dr Genevieve Hayes: So what exactly is involved in your role as head of research at Gradient?
[00:03:12] Kyle Block: As the head of research, my role is to attempt to bridge what is occurring in the methodological best practices that Primarily academic researchers are bringing to the fore and ensure that those are applied and infused into our more market based applications of survey research. And I have the good fortune of being able to develop a research vision or an approach for pretty much all of our projects.
[00:03:47] Each of them is unique, and I get to set up a framework, the type of analysis, and help our clients and my teammates think about how we're going to use the survey data that we ultimately collect and summarize it. in ways that are actually useful to help our clients make what are almost always very hard decisions.
[00:04:12] Dr Genevieve Hayes: I like that term survey vision. Was it survey vision or research vision you just said?
[00:04:18] Kyle Block: Good question. I think both could work. They're deeply related, but the research vision refers to kind of the big picture. How are we going to get from an important strategy question to the specific statistic that is going to help you make that decision? And then the survey vision refers to how we're actually going to measure and collect in a real survey that individuals are going to respond to.
[00:04:43] Dr Genevieve Hayes: So it's sort of like a roadmap for the work ahead of you,
[00:04:47] Kyle Block: That's a great way to frame it, yeah.
[00:04:48] Dr Genevieve Hayes: And would that just be a one pager or would it be, you know, a small novel?
[00:04:53] Kyle Block: We at Gradient when every project kicks off, we prepare no offense to any listeners here, what we call a project bible. And this is a usually three to four page authority. on the vision for the project to be executed. And it starts with outlining at the very first line, what is the strategy question that this research needs to answer?
[00:05:23] And then from there, then we will consider what possible statistics could we come up with to help answer that question. And then we go to the next phase, which is to contemplate How could we get variables to be able to analyze so that we could get to that statistic? And so that we all outline in that project Bible, and once that is agreed upon both internally and with our client partners, then and only then do we actually start writing a survey instrument and programming the survey and contemplating how we're actually going to write the code to analyze it.
[00:06:02] But we start with that vision. And we spend a lot of time on it, and then and only then will we actually start building something, if you will.
[00:06:11] Dr Genevieve Hayes: And can you give us some examples of some of the research questions that you might come across on a typical day at Gradient?
[00:06:18] Kyle Block: Certainly, there's a category of questions that we frequently answer related to defining an audience. What is an audience? What are the elements within that audience that are perhaps more profitable or more likely to consider converting to A new product or would be very averse to converting. So there's questions around audience and defining them.
[00:06:45] A lot of other questions that we help answer relate to brand or organizational level questions such as, How is my brand or my organization perceived? Or what are the brand attributes or equities as are sometimes known that are actually what drive value that attract people. And this could be buyers, this could be potential employees to actually value the brand.
[00:07:11] And then the other category of questions that we help answer relate more to products. And these could be questions like, what are the product features that my current customers actually want to see in the next release? Or it could be a little more complex. It could be what combination of product features at a certain price point in a particular market for a very specific audience segment is optimal.
[00:07:37] So we, just to summarize, answer questions related to audience, brand, organization, and product. And that makes for a very interesting day job, if you will.
[00:07:51] Dr Genevieve Hayes: Do you just work in the marketing sphere or would you also apply these skills to other areas like say understanding people who'd be interested in voting for a particular political candidate cause I can imagine there's a lot of political applications for this sort of work?
[00:08:08] Kyle Block: There certainly is, and we do a lot of political oriented or socially oriented applications of survey research also. So we do help some of our clients understand what is the appetite of voters for change on a specific issue. We work with politically oriented clients who want to understand how can they best message To a particular audience.
[00:08:35] And we use survey methods to do that. And then we also have the good fortune of working with philanthropies who have big ambitions to change what seemed like intractable problems in American society. And we get to help them answer really, really big questions. Like what does the average American really think success looks like?
[00:09:01] What are the constituent parts of success? How are those different from what society tells the average individual? So those are more of the social and political applications of survey research that we get to work with on a daily basis.
[00:09:17] Dr Genevieve Hayes: Out of interest, what does the average American think that success looks like?
[00:09:22] Kyle Block: I'm so glad you asked, but I'm going to answer a slightly different question first, because as I mentioned, we measured not only what does the average individual think success looks like, but we also measured what does the average person think society's Definition of success looks like, and we broke it up into like 70 atomic particles, all of which could ladder up to success and society's definition of success is almost entirely made up of being famous.
[00:09:58] That's it.
[00:09:59] Dr Genevieve Hayes: So being Kim Kardashian or something.
[00:10:02] Kyle Block: Exactly. And that is what we are told success is. Do you want to guess where that particular element falls on the individual definition of success?
[00:10:13] Dr Genevieve Hayes: I suspect it's relatively low in the list of priorities
[00:10:17] Kyle Block: Exactly. It's the least important. It's at the very bottom. So we've got a complete inversion between what people, real individuals, actually want their lives to be and what our society is telling them. And then the other beauty is that the individual definition of success, there really isn't one. It's so individualized.
[00:10:40] Everyone has a different opposition and that was a really profound finding from that work.
[00:10:46] Dr Genevieve Hayes: I think this is actually a really good case study for how would you design a survey that could find out what you just told me?
[00:10:56] Kyle Block: There was no playbook to follow and we had to do a lot of experimentation to find the right method that could be used to answer such an amorphous and comprehensive question. So we actually started with kind of the both possible best companion and enemy of a survey, which was qualitative research.
[00:11:24] We did a lot of listening. We spoke with a broad spectrum of Americans and asked them very pointed questions around their definition of success. And we listened, we cataloged a lot of that, and they gave us some inspiration for how we might need to start conceptualizing. How we would measure success. And then after that qualitative phase, we prepared three different survey based methods that we thought might be able to measure success.
[00:11:55] And we compare the results of three of them. And it really was only one that made sense. That was practical that revealed the exact findings I shared earlier and. Not only did they make sense, that's sort of critical in this kind of work, but we were able to summarize them in intuitive ways by being able to say that fame constitutes, you know, 90 percent of society's success and, you know, less than 5 percent of the individual, the way in which we were able to present a statistic.
[00:12:27] It's just as important as how we calculate it, but that explanation factor is really critical and so the method we landed on at a higher, if you will, kind of intuitive understanding in trying to explain what it is that we actually
[00:12:43] Dr Genevieve Hayes: When you're talking about the different methods, are you referring to differences in the actual questions themselves or the type of question? So for example, by type of question, I mean like arts scale versus ranking versus, I don't know, whatever other type of survey question you have, or is it how you were phrasing the question that made the difference.
[00:13:08] Kyle Block: in most cases, what we're trying to examine is the methodological difference. So. Assuming we're going to keep the question wording more or less similar across methods, which we felt comfortable doing, what's really interesting us is the way that you try to measure a particular phenomenon. So you're exactly right.
[00:13:30] Like, in some cases, a Likert scale is very appropriate to measure a specific reality, but not for all of them, particularly for something that cannot be easily categorized into three to six discrete categories. Like, how does one define success? So it's those kinds of methodological norms that we're trying to evolve and where survey research I think needs a little nudge, is in identifying More novel ways to measure important constructs.
[00:14:02] So one of them is, you know, the Likert scale is probably the most common survey methodology and it does a good job where it can shine, but it's not right to answer questions like, how do you define success? And so that's what we try to develop at Grady. There's a whole new.
[00:14:21] methodology to be able to capture the phenomenon we're trying to measure.
[00:14:25] Dr Genevieve Hayes: I'm having a difficult time imagining a type of survey question beyond Leichhardt scale or ranking preferences type questions. Can you give me an example of what this might look like?
[00:14:39] Kyle Block: I'd love to. I'm so glad you asked. And we hope to get many more questions like that because you're certainly not alone, right? Most people, even an experienced academic no offense to the academic world, are just very familiar with a few of the possible types of survey methodology. So one area that we've been Developing a lot of methods around is a discrete choice experiments.
[00:15:06] So these are survey based methodologies, but they look a little bit different, both from the respondent and analytical perspective, then your standard, you know, tell me what your most preferred color is right. So just be choice experiment. Makes an assumption that any decision most consumers or voters or business leaders need to make has some trade off, and the discrete choice experiment forces respondents to actually trade off something in expressing their preference.
[00:15:45] So one of the most I think underutilized survey methodologies that is a discrete choice experiment is a conjoint analysis. So conjoint is kind of a combination of considered jointly, and this is as close to as we feel like we can get in a survey based environment to simulating what a consumer's real life decision making looks like.
[00:16:13] So I'll paint a picture really quickly from a respondents perspective, what participating in a conjoint looks like. Let's say a car manufacturer is trying to figure out a new model and the questions that they're trying to deliberate on are what color should this model be?
[00:16:32] What type of transmission should it have? What type of entertainment system should it have? And should it be a, fossil fuel powered electric or hybrid? And so for each of those attributes, there's going to be different options, right?
[00:16:44] For color, you have red, green, blue, different types of transmission. So respondents are going to see when they log into the survey, they're going to see two side by side profiles. On the left, it'll have a different random combination of product features. Red, automatic, upholstery. And on the other column, it's going to have a different random combination, blue leather seats, maybe a CD player, if this was like 2002, and a price for each of them.
[00:17:16] And the respondents are going to determine which of those that they prefer the most. And then they're going to click next. And a completely different random combination of two different profiles is going to appear. And they're going to make the same choice. And they're going to do that about 10 times.
[00:17:32] And then what we can do with that data, which produces like a very wide matrix of correlations, we can develop a model that can tell us precisely the extent to which color plays a role in determining a consumer's preference. And Which color is most preferred. And we can do that for all of the attributes that were tested.
[00:17:56] And so this is a survey based method. It's a little more involved, but it gives you very, very precise and actionable results that you can take to your design team and say, here's what we need to build, I'm searching about this, let's go in
[00:18:10] Dr Genevieve Hayes: Would every person who did that survey get the same combination of random attributes for each car? Or if I did the survey, would that be different from someone else doing the survey?
[00:18:24] Kyle Block: most designs. Every single. experience is going to be unique. And that's how we can sort of make all of these different combinations possible and back out what is actually driving one's preference.
[00:18:38] Dr Genevieve Hayes: Yeah, I thought that would have to be the case because otherwise if you had every single combination, each individual would have a prohibitively high number of Choices to make and they'd get bored and they'd just start clicking at random. But if you had, say, a thousand people and each of them got a different set, you could give them a smaller number of choices to make.
[00:18:59] Is that right?
[00:19:00] Kyle Block: It's exactly right.
[00:19:02] Dr Genevieve Hayes: And then no one actually wants to walk out of the survey.
[00:19:05] Kyle Block: Yeah. Which is a whole other complication that a good survey design needs to consider, but we could talk about that next.
[00:19:13] Dr Genevieve Hayes: Yeah. It's actually something I do want to discuss. So I think we've all come across. Bad survey design. I mean, pretty much anything on Buzzfeed is an example of bad survey design. I came across a Buzzfeed quiz the other day that claim to be able to predict whether you were British, American, or Australian based on your chicken shop preferences.
[00:19:38] So if you went into a chicken shop, which of these different types of chicken would you want? You know, chicken nuggets, a rotisserie chicken, all this. And I was bored. It was a Friday. I had nothing to do. So I thought, okay, let's give this a go. And it predicted I was British, even though I've never set foot in the UK.
[00:19:58] And yeah, so I think, I would
[00:20:01] Kyle Block: Law of survey design.
[00:20:03] Dr Genevieve Hayes: I suspect it has no statistical validity whatsoever.
[00:20:08] Kyle Block: I'm going to go with highly unlikely.
[00:20:10] Dr Genevieve Hayes: Yeah. But what does good survey design look like?
[00:20:15] Kyle Block: Good survey design. Is one of those areas that I think most people are surprised actually is hard to do right. You think BuzzFeed can design surveys and they feel them people answer them and they publish them. But there is a science to survey design. And there's I think, two elements that constitute good survey design.
[00:20:42] And one is, of course, related to the survey and how it's constructed, how it's phrased. Is it developed with an inclusive understanding of how every single person is going to interpret the questions that are asked? In the way that you as a researcher want that question to be interpreted, and if that is not true, then your survey is going to be rife with what's known as measurement error, which simply means you're measuring something that you're not intending to, but you're claiming to have found something.
[00:21:23] And it's just, in fact, inaccurate. So there are a variety of ways to avoid measurement error. And one of them does rely upon a qualitative technique called a cognitive interview, which is essentially a live synchronous survey interview with a participant, usually in a laboratory setting, and the researcher, where the researcher walks through a draft survey and poses a lot of very specific probes To ask the respondent.
[00:21:59] How did you interpret this or how did you arrive at your answer? And the purpose of this exercise is to break down every single cognitive step that a respondent must go through to read your survey question, evaluate the possible answers, contemplate if they even understand it, evaluate how they're going to answer it.
[00:22:24] And without knowing those cognitive pathways, You're highly likely to design this survey question, let alone a whole survey. That is going to be full of measurement error and a cognitive interview will show you where you're accidentally imposing perhaps confusing or not inclusive question wording response options.
[00:22:50] That you have to fix before you can competently field your survey.
[00:22:55] Dr Genevieve Hayes: I understand that. My family does the newspaper quiz every evening. And there've actually been quite a few over the years where we realize, after we've seen the answers that we were answering a completely different question from what the person who wrote it actually had in mind.
[00:23:16] So, I'm guessing that sort of thing would happen over and over again in surveys.
[00:23:20] Kyle Block: Exactly. Again, it's easy to make that mistake, but it is a really fatal one, particularly when you're going to use the results from that survey to decide what audience to go after, how to price your product, how to talk to your own employees, if you're a CEO of a massive organization, so.
[00:23:40] It really does matter that you can be certain that you've measured what you have intended.
[00:23:46] Dr Genevieve Hayes: The title of this episode is The Art and Science of Survey Design and I expected there to be more art in it, but, just what you're saying there, it's a very scientific process to how you write the questions.
[00:23:58] Kyle Block: It sure is. And I hope that if there's any lesson imparted from our time today, that listeners do recognize there is a science to survey design. And even early in my career, I didn't necessarily realize that, but I found a couple universities that specialized in survey methodology, and there's just a handful of them.
[00:24:21] But this is where, you know, individuals who design national census. and statistics analyses get their training because it's really important.
[00:24:30] Dr Genevieve Hayes: Yeah. Based on my own experience one of the hardest things to research from survey data is people's opinions around sensitive or controversial subjects, even if the survey is anonymous. For example, with those telephone polls, where they ask you who are you voting for in the next election, you'll have some people who just hang up the phone, others who deliberately say the other party just to.
[00:24:56] Mess up the results. And then you've got the sorts of surveys that are asking people's views around something illegal, like cheating on your taxes. I mean, if someone asks, have you ever cheated on your taxes? How many people are going to say, yes, I do it every year and I'm planning on doing it next year too.
[00:25:15] That is never going to happen. But you've come up with a way to get around this issue, haven't you?
[00:25:21] Kyle Block: We have. And it's been a multi year scientific, and yes, that can be true even in surveys development. Because I of course love working with survey research, but one of its flaws Is, as you've astutely noted, it may not always be revealing the truth. And so, as a survey scientist, I'm trying to find the truth in what a population believes or feels or perceives.
[00:25:49] And that was very unsettling to know that even if we designed the survey correctly and has the perfect sample frame, it still may not reveal the truth. So, we did some exploration and found, of course, that there were some questions around the veracity of answers to sensitive statements, as you noted. And we looked back in the academic literature and found some researchers in academic institutions who had suggested kind of some lightweight frameworks for other ways to try to solicit perceptions or behaviors on sensitive attitudes.
[00:26:29] But there was no guide, there was no definitive, this is the way to do it. And so we've developed that over the past couple of years. And it uses what's called the list experiment or the item count technique as the workhorse of the methodology. And over the past couple years, we've just seen a very, very consistent trend that using this list experiment methodology is able to identify in a broader population where there is widespread differences.
[00:27:03] Between what public opinions that's what people will respond to in a standard all reveals relative to what private the they feel on the same issue.
[00:27:18] Dr Genevieve Hayes: So how do these list experiments work?
[00:27:20] Kyle Block: Well, it's not terribly different from the discrete choice experiment in that it's another kind of experimental method that we use a survey. To bring to life. So imagine. That we have some suspicions that more Americans are cheating on their taxes than they're likely to admit to so very sensitive, right? You're not going to go up to your neighbor or probably even a family member and admittedly say, Guess what?
[00:27:54] I did. I cheated on my taxes, but the IRS Or a tax authority anywhere in the world would probably have an interest in knowing if there at least is some semblance of a difference between what people are publicly saying and maybe what they're actually doing. And so the list experiment is a perfect kind of Oracle to develop this.
[00:28:18] So the way that it works is we're trying to test the agreement with the statement. I have cheated on my taxes. In the past. And what we do is we develop a survey and half of the respondents in the survey get assigned to a control condition and half get assigned to a treatment condition in the control condition.
[00:28:44] Respondents will see on their screen a list of four seemingly unrelated statements. It could be statements like I I'm kind of happy with my life today, or I'm kind of anxious about my finances for tomorrow. I think the cost of college is too high and inflation is hurting my bottom line. So that's what the control condition sees.
[00:29:13] And the question that the respondent in the control condition is asked is, how many of these statements do you agree with? Not which ones, but how many, and then the other half of the survey sample that's in the treatment condition, they'll see the same four statements in addition to the fifth sensitive statement being, I have cheated on my taxes, and they're asked the same question, how many of these apply to you.
[00:29:44] And if we see a statistically significant and large difference. between the numbers that are reported between the treatment and control conditions, then we can say that there is a possibly, if it's true, a high probability that there is a preference falsification occurring and people are hiding their true beliefs on these sensitive items.
[00:30:14] Dr Genevieve Hayes: So the probability part of this, this would just be a straight application of Bayes rule, wouldn't it?
[00:30:19] Kyle Block: Yeah, that's exactly right.
[00:30:20] Dr Genevieve Hayes: Yeah. And then you just work out whether there's a statistically significant difference. Yeah.
[00:30:26] Kyle Block: Yeah, and so there's frameworks, there's tools out there that make this work. We've had to do a lot of tinkering to understand just like you're going to design a good public opinion survey. There's certain best practices with how you phrase statements. For example, and how correlated the non sensitive statements need to be with one another that we've had to investigate and identify over the past couple of years, because that was not well established in the field.
[00:30:57] But now we've done a lot of methodological testing to be able to say with confidence, there is a right and a wrong way to use list experiments, and now we can do that with a lot more confidence.
[00:31:07] Dr Genevieve Hayes: Yeah, because I could imagine you could run into problems if the four non controversial statements were ones that everyone agrees on.
[00:31:16] Kyle Block: Exactly.
[00:31:17] Dr Genevieve Hayes: Because if everyone's giving four out of four, then I would say that people would be reluctant to say, I agree with all of these statements if one of them is, I think cheating on my taxes is a good idea.
[00:31:30] So if you had four non controversial statements, you'd want, on average, people to disagree with at least, say, two of them, so that you can then hide the agreement with the dodgy one.
[00:31:43] Kyle Block: Exactly. And so one of the initial phases that we have to go through before we can even field the actual LISP experiment survey is to build another survey to identify the correlation between the non sensitive statements so that we can create bundles of what we call control items or the non sensitive statements.
[00:32:05] So that we can assure that there's some anonymous hidden likelihood that they won't possibly agree with all four.
[00:32:13] Dr Genevieve Hayes: I'm just trying to think, what are two statements that have a very high negative correlation with each other, because I just can't think of any unless you're having literally the opposite of a particular statement.
[00:32:26] Kyle Block: That's a good question. So it could be things like you know, I'm not concerned at all about poverty and what would be the inverse of that? Like some, some notion of, you know, their own personal wealth or. That's a good question. And they don't have to be completely.
[00:32:44] Inversely related, but they need to have some kind of space between them.
[00:32:50] Dr Genevieve Hayes: I guess you could have a bad thing and it would be nice to help. homeless people get homes versus I don't give a damn about poverty as long as I have more money in my pocket
[00:33:01] Kyle Block: exactly.
[00:33:02] Dr Genevieve Hayes: How big would your sample size have to be in order to get meaningful results out of this sort of experiment?
[00:33:08] Kyle Block: Sample sizes for this experiment, as you can probably imagine, do need to be high. And by that, I mean, relative to a similar sample size to measure public opinion, because there is a lot more noise, in what you're introducing both for the respondent experience and the modeling. So for every kind of subgroup, you do want to be able to say there is some level of difference between public and private opinion.
[00:33:39] You need to have, somewhere between 500 to 1000 respondents for each subgroup, but typically for a standard public opinion poll for an entire population, 1000 respondents is usually good enough with a reasonable margin of error. So that is one of the drawbacks to this experiment is it does increase your sample size and therefore data acquisition costs
[00:34:04] Dr Genevieve Hayes: And do you have to have equal numbers being in the control group and the group that gets the controversial statement?
[00:34:10] Kyle Block: ideally. Yes.
[00:34:12] Dr Genevieve Hayes: I can imagine how these sorts of surveys would come up a lot in, say, political surveys and surveys that might be done by charitable organizations. Do they come up very often in a marketing situation?
[00:34:26] Kyle Block: They're just beginning to, and this is a very exciting kind of frontier of where we think the whole category of private opinion could be really valuable in a world where there are so many surveys. There's so much that we're told about other people from our media, from our friends that I think generally a lot of people are not sure what to believe about those around us.
[00:34:53] Dr Genevieve Hayes: Really?
[00:34:54] Kyle Block: Yeah, and we've gotten evidence to suggest this from years of this list experiment research. And so we're seeing that there is a lot of value in using it in more commercial applications. So one really prominent example is a lot of business leaders after the pandemic have been clamoring to get their employees back in the office.
[00:35:18] And the public opinion data has been overwhelmingly clear that most employees do not want to go back to the office, at least not five times a week. We have also heard from business leaders that they're like, I'm just not sure this is true. They're suspicious that that public opinion data that's reported is like self reinforcing.
[00:35:43] As people get further away from having social interactions and a sense of community, but then keep being told by the media and their colleagues that I really don't want to go back to the office. It does, even an anonymous public opinion survey change what you'll say.
[00:36:02] So, one of the probably. Most actionable applications of the list experiment is to help business leaders identify with precision, the extent to which their employees do actually want to come back into the office or maybe feel uncomfortable working from home or feel disengaged from the workplace because they don't feel comfortable saying this in a public opinion setting. This is really critical for business leaders to understand because if they're responding to the wrong signals, then they're just going to aggravate their employees even more. But the signals that they're responding to need to be a private one and not the public ones because then they're going to make the wrong decisions.
[00:36:49] To make private opinion even further away from public opinion. And when that is the reality that a leader is up against, there's really no way to kind of stop that fire once it started.
[00:37:01] Dr Genevieve Hayes: Oh, that's interesting. Because what I was thinking about, this is going back to the political case, with. Our last election in Australia, there was commentary coming out around the results in the electorate that I'm in. And one thing I found really fascinating was that what the media was interpreting had nothing to do with what was actually the case in the electorate, they completely got it wrong.
[00:37:26] They were saying, it was due to reason X when it might have contributed to it, but there was a lot more to it than that. And in an office environment, there'd be things where there's the headline reason that gets reported in the media.
[00:37:42] And then there's the, if you actually go and speak to the actual people on the ground, there's a lot more complications than that.
[00:37:49] Kyle Block: Exactly. Yeah. And that's not a, a new reality, but it's never been really possible to measure in a generalizable sense what people on the ground, really truly feel. Until you have a coffee with them, or a drink or two. That's not a sustainable practice, but we need to know what the people on the ground really feel.
[00:38:11] So the excitement of this private opinion methodology is that we can generalize the extent to which those are distinctions that are potentially very large and would cause leaders to make probably very different decisions if they knew the people on the ground really truly felt.
[00:38:29] Dr Genevieve Hayes: So far we've been focusing largely on the design side of survey research, but that's just one half of the equation. Once you've designed your survey and collected your data, to get value from that data, you need to go off and analyze it. Now, most people when analyzing survey data will just produce a ton of charts showing the proportion of people who gave each result.
[00:38:54] SurveyMonkey does that automatically, but That always seemed very simplistic to me, you've already mentioned using Bayesian techniques in order to analyze that. List experiment. What other ways are you at Gradient using advanced statistical techniques and machine learning to get value from the analysis of your survey data?
[00:39:18] Kyle Block: I really appreciate the question. Because I do think there is a bit of a misconception that survey data by the nature of how it's structured is simple to analyze, and it can be if your definition of analysis is a series of cross tabs, which, yeah, SurveyMonkey and everybody else can do that automatically and it's interesting, but I would say not even necessarily informative if that is the end game of your analysis.
[00:39:49] So those are descriptive statistics. Our paradigm at gradient is to use the exact same output that you would Get from a survey. But before we even have put survey questions on paper, we are contemplating what kind of statistical model could be used to extract more information, more clarity, more precision from survey data that doesn't require one to look through rows and rows and columns and columns of cross tabs and essentially cherry pick what is a value or what they're going to react to.
[00:40:29] Because if you're doing that, then you're looking for, a cell and maybe two or three cells that are statistically significantly different from the population mean, and you can claim to have found an insight. But if there's a fourth value you just didn't scroll far enough to see, that could completely change the insight and the recommendation that's going to come from it.
[00:40:49] So it's a very, kind of crude deduction of Analysis, but we're trying to avoid that completely by using model based analyses that have a couple benefits, one of which is. The models can take as an input much more data and more variables, more options and take a more bottom up approach to identify what could be driving a particular finale.
[00:41:15] So that's one. We always want to think about our preference for a model based analysis, and then we'll design the survey. So that it can very precisely produce the exact kind of data structure to work with the model that we've contemplated a priori. And then the other, and you already hinted at this, which I'm very grateful for design element that we've taken to account is how are we going to summarize and ultimately visualize the outcomes of our analyses.
[00:41:48] But the last thing we want to do is stare at a spreadsheet. That looks a bit like a checkerboard with red shading and green shading because you can just tell your own story, pick your own adventure, and that's not exactly the kind of precision I think anyone should be aiming for. In a model based world, though, you can get covariates and measures of accuracy and probabilities that can tell you.
[00:42:12] You know, if you're trying to measure the favorability of a particular brand and what's driving that a model is much more likely to tell you a revealed reality that it is a particular attribute of a brand that is contributing to its favorability. That in a descriptive analysis may not actually reveal because it's analyzing everything in a vacuum.
[00:42:35] Dr Genevieve Hayes: By models, are we talking about statistical or machine learning models or are we just talking about very simplistic spreadsheet type models? Because I've heard people use the term model to describe both of those.
[00:42:49] Kyle Block: Great question. I'm referring to statistical models where there are, you know, Bayesian, so a spreadsheet couldn't handle this. So we work primarily in R and state. And the models we use run multiple simulations. We compare a lot of different types of models and model specifications using, of course, the same data sets.
[00:43:11] So we can have confidence in what we're ultimately identifying
[00:43:15] Dr Genevieve Hayes: Okay. So it might be a linear regression or logistic regression or something. Yeah. Okay.
[00:43:20] Kyle Block: Yeah. Yeah. Could
[00:43:21] Dr Genevieve Hayes: So that makes sense. So it is quite statistical in nature.
[00:43:25] Kyle Block: It's very sophistical.
[00:43:26] Dr Genevieve Hayes: Yeah. And fact that you're using R and Stata, that, really tips off. Yeah. You're going into the hardcore stats.
[00:43:34] Kyle Block: There we are. Yep.
[00:43:36] Dr Genevieve Hayes: Yeah. What's your go to statistical technique that you find comes up again and again?
[00:43:43] Kyle Block: We have over the years been quickly evolving to using a Bayesian. Orientation for every model that we design
[00:43:51] Dr Genevieve Hayes: It doesn't surprise me.
[00:43:52] Kyle Block: if you have for a lot of reasons, but one of which is even just the interpretation is a little bit easier for some of our less academically trained clients to understand it's harder to develop and run and the computational needs are intense, but the interpretation we've been able to say as a probability has been good.
[00:44:16] Our distribution has been easier to convey to many of our clients.
[00:44:20] Dr Genevieve Hayes: Are there any techniques that you haven't had the opportunity to apply to survey data, but that you think hold promise or that you'd really like to apply?
[00:44:30] Kyle Block: One area where I think there's still some room for investigation is how respondents reply to pricing in a survey. Cause it is abstract, but there's not a product in front of them. They're not actually going to swipe or click. And , my honest truth is, I don't know if this is It's a question that fits more in the survey science realm or the survey art, but it is, I think the next R and D effort that we're going to pursue is to get smarter on what is cognitively occurring when respondents are responding to pricing related questions in a survey environment.
[00:45:13] Dr Genevieve Hayes: Yeah. Okay. And that makes sense. In Australia at the moment, we've got a cost of living crisis and I don't know if it's the same that you're experiencing where you are, but it's become such a sensitive topic. I can see that there would be a lot of demand for that in survey research.
[00:45:32] Kyle Block: Yeah. And a lot of the reactions to these crises are after the fact, right? Prices have already gone up on shelves. And the ultimate benefit of survey research is you can learn a lot of things about hypothetical reactions before they may actually become reality. Right. A brand can say, well, I'm going to test this messaging construct.
[00:45:56] I'm going to test this different pricing scenario before I actually spend the millions of dollars to implement it. But we need to be certain in what we're measuring. So we're looking forward to getting a lot smarter on how respondents evaluate types of elements in a survey.
[00:46:11] Dr Genevieve Hayes: What final advice would you give to data scientists who are looking to create business value from data? Yeah. So
[00:46:19] Kyle Block: parting piece of advice I would offer is survey design is of course critical, but it is one of two essential elements to being able to confidently use survey research in making hard decisions. And that second piece is One must also understand where those survey results are coming from. In other words, that you know that your sample is representative of the population you're trying to actually make generalizations about.
[00:46:53] And this probably seems like common sense, but even to very seasoned marketers who want to identify the growth opportunity. In their market. Frequently, I have to make the case to say, I hear you and we want to help you with that. But to be able to say anything about the growth opportunity means you need to have a sample that includes customers who are not currently in the industry.
[00:47:22] Your own. And they say, Oh, but we always just sample from our CRM system. And I say, Great. And you're learning all the about your current customers and not the opportunity. So I'd say think inclusive and broadly because that is where a lot of the learning needs to happen. But where your sample and survey data comes from.
[00:47:43] Is a fundamentally important question that you should have a very strong point of view on as you're designing your overall survey project.
[00:47:52] Dr Genevieve Hayes: for listeners who'd like to learn more about you or get in contact, what can they do?
[00:47:57] Kyle Block: You are more than welcome to email me, Kyle, KYLE, at gradient metrics.com. If you would like to hear more about the latest in survey developments and methodological best practices. We have a lot of content that we share on LinkedIn. So Kyle block at gradient, and you can learn a lot more about our developments in the survey realm on LinkedIn.
[00:48:23] Dr Genevieve Hayes: And Gradient also has a newsletter, and that has some very interesting articles in it about things that you've discovered in your surveys.
[00:48:32] Kyle Block: Yeah. Trendlines is a fun newsletter that we prepare. It's all features, surveys that we fielded. That we think are interesting that use methodologies that are unique to gradient. And we think we presented in a bit of a humorous, sassy way. And so it's not just dry charts and stats. There's a little bit of humor mixed in there too.
[00:48:54] So that's trend lines. You can subscribe at the gradient metrics websites. If you're curious and want to maybe laugh a little bit,
[00:49:03] Dr Genevieve Hayes: It's a good fun way of learning about statistics, I think. So
[00:49:08] Kyle Block: Very generous of you.
[00:49:10] Dr Genevieve Hayes: thank you for joining me today, Kyle.
[00:49:13] Kyle Block: Thank you, Genevieve. This was so fun. I wish you all the best and thank you for sharing the story with your listeners.
[00:49:20] Dr Genevieve Hayes: And for those in the audience, thank you for listening. I'm Dr. Genevieve Hayes, and this has been Value Driven Data Science, brought to you by Genevieve Hayes Consulting.
Creators and Guests
