Episode Transcript
[00:00:00] Speaker A: Welcome back to the Deep Dive.
[00:00:01] Speaker B: Glad to be back.
[00:00:02] Speaker A: Today we're diving into a topic I think a lot of our listeners are already pretty familiar with. AB testing.
[00:00:08] Speaker B: Yeah.
[00:00:09] Speaker A: But we're going to be looking at it with a new lens today through the power of AI.
[00:00:14] Speaker B: Right.
[00:00:14] Speaker A: And how it can take your AB testing analysis to a whole new level.
[00:00:18] Speaker B: Yeah. And really kind of get you to a point where you're making smarter decisions.
[00:00:22] Speaker A: Exactly. Yeah. Get better results from your marketing campaigns.
[00:00:25] Speaker B: Smarter decisions, better roi.
[00:00:27] Speaker A: Yeah. And I think that's something we all, as sales and marketing executives are always striving for. Right, Definitely. But, you know, it's funny because I think a lot of us like to think that we have a pretty good intuition about what works in marketing.
[00:00:40] Speaker B: You think you have good grasp on it?
[00:00:42] Speaker A: Yeah, I trust my gut. Right. Been doing this for a while, so I kind of know what's going to work and what's not going to work.
[00:00:47] Speaker B: Yeah.
[00:00:47] Speaker A: But I think what's really interesting about this conversation is that sometimes that intuition can lead us astray.
[00:00:54] Speaker B: It can, Absolutely. Um, especially when there are these hidden factors at play that we might not even be aware of, that we don't.
[00:01:01] Speaker A: Even know are there.
[00:01:02] Speaker B: Right, Exactly. And that's where I think the power of AI and using that data driven approach can really come in.
[00:01:08] Speaker A: Yeah. So you're saying that my gut isn't always enough?
[00:01:11] Speaker B: It's not always enough, no.
[00:01:13] Speaker A: Okay, I'm, I'm intrigued. So can you give me an example of where our intuition might mislead us?
[00:01:18] Speaker B: Absolutely.
I think a fascinating example, and it might seem a little unrelated at first, but the principle applies directly to marketing is an example from the medical field actually called the smoker paradox.
[00:01:31] Speaker A: Smoker paradox.
[00:01:32] Speaker B: The smoker paradox.
[00:01:33] Speaker A: I've never heard of this.
[00:01:34] Speaker B: Yeah. So there was a study that was done on a lung cancer drug, and initially it seemed like the drug might actually be harmful because the people who received the drug had a higher mortality rate compared to those who didn't.
[00:01:46] Speaker A: Wait, so the drug was actually making things worse?
[00:01:49] Speaker B: That's what it seemed like. Right.
[00:01:50] Speaker A: Not exactly a ringing endorsement.
[00:01:52] Speaker B: Not a great selling point if I'm a patient. Exactly. But here's where it gets interesting is when they dug deeper and they realized that the group receiving the drug had a much higher percentage of smokers compared to the control group.
[00:02:05] Speaker A: Ah, okay.
[00:02:06] Speaker B: So they were already at higher risk.
[00:02:08] Speaker A: They were already predisposed to potentially having a negative outcome regardless of the drug. Regardless of the drug.
[00:02:14] Speaker B: That makes sense.
[00:02:15] Speaker A: So really what the researchers were doing is they were comparing apples to oranges. And this is where that careful ab testing analysis comes in, is by adjusting for those differences in smoking habits, they were able to see the drug's true effectiveness, and it turned out to be beneficial, just masked by those initial misleading numbers.
[00:02:36] Speaker B: Interesting.
[00:02:37] Speaker A: Yeah.
[00:02:37] Speaker B: So it's almost like those hidden differences were creating an illusion.
[00:02:41] Speaker A: An illusion? Yeah.
[00:02:42] Speaker B: Making it look like the drug was the problem when really it wasn't.
[00:02:45] Speaker A: Yeah. It was those other underlying factors that were really driving that difference.
[00:02:49] Speaker B: Okay. And I can already see how this might apply to marketing as well.
[00:02:53] Speaker A: Oh, absolutely.
[00:02:54] Speaker B: We're always trying to figure out what's causing what.
[00:02:56] Speaker A: So can you give me a marketing example?
[00:02:58] Speaker B: Yeah. Let's say you're a telecom company.
[00:03:00] Speaker A: Okay.
[00:03:01] Speaker B: And you're trying to figure out if promoting online backup services helps with customer retention.
So you run an A B test, and at first glance, it looks like the backup service is a win. Customers who have it are churning at a lower rate.
[00:03:16] Speaker A: Sounds like a success story.
[00:03:17] Speaker B: Sounds great, right?
[00:03:18] Speaker A: I'm already ready to roll this out company wide.
[00:03:21] Speaker B: Yeah, but there's a twist.
[00:03:22] Speaker A: But there's a twist. I should have known.
[00:03:24] Speaker B: Always a twist.
[00:03:25] Speaker A: It's always a twist on this show.
[00:03:27] Speaker B: And the twist is that the group who opted for that backup service tended to have certain characteristics in common. Things like being older, married, being on annual contracts. And guess what? All of those factors are already linked to lower churn rates.
[00:03:42] Speaker A: Okay. So once again, it wasn't necessarily the backup service itself.
[00:03:46] Speaker B: Not necessarily, no.
[00:03:47] Speaker A: It was other underlying factors.
[00:03:49] Speaker B: Other factors driving the results.
[00:03:50] Speaker A: Right. And without accounting for those hidden variables.
[00:03:53] Speaker B: Right.
[00:03:54] Speaker A: You might think your marketing efforts are more effective than they actually are. You might even be making decisions that are hurting your business.
[00:04:01] Speaker B: You could be making decisions that are detrimental to your business without even realizing it.
[00:04:05] Speaker A: That's a scary thought.
[00:04:06] Speaker B: It is scary.
[00:04:07] Speaker A: So how do we avoid falling into this trap? How can we make sure that we're seeing the real picture?
[00:04:12] Speaker B: Yeah, that's where AI comes in. And using techniques like propensity score matching, we can create a virtual control group that mirrors those key attributes of the treated group, giving us a much more accurate comparison.
[00:04:26] Speaker A: Okay. This is where my brain starts to hurt a little bit.
[00:04:28] Speaker B: Yeah.
[00:04:28] Speaker A: Can you break that down for me? What is propensity score matching?
[00:04:32] Speaker B: Yeah, so propensity score matching is essentially. Imagine we could magically create a control group that has the same mix of ages, contract types, product usage as the group who got the marketing campaign.
[00:04:46] Speaker A: Okay.
[00:04:47] Speaker B: That's what propensity score matching is doing, is it's? Helping us to create that so that we're making it a much more apples to apples compared.
[00:04:53] Speaker A: So we're leveling the playing field.
[00:04:55] Speaker B: Leveling the playing field. And by doing that, we can get a much clearer picture of the true impact of our marketing efforts.
[00:05:01] Speaker A: So in the telecom example, once they applied these techniques, what happened?
[00:05:06] Speaker B: Once they applied these techniques, they discovered that the backup service was actually increasing. Churn.
[00:05:11] Speaker A: Whoa.
[00:05:12] Speaker B: Yeah.
[00:05:13] Speaker A: A complete 180.
[00:05:14] Speaker B: A complete 180.
[00:05:15] Speaker A: What seemed like a win was actually having the opposite effect.
[00:05:19] Speaker B: Exactly.
[00:05:20] Speaker A: That's pretty mind blowing.
[00:05:21] Speaker B: It is mind blowing. And without that deeper level of analysis, they would continue pushing that backup service, thinking they were doing the right thing, when in fact it was hurting their customer retention.
[00:05:30] Speaker A: Yeah, they would have been doubling down on a losing strategy, essentially.
[00:05:33] Speaker B: Exactly.
[00:05:34] Speaker A: Okay, I'm officially hooked. This stuff is fascinating.
[00:05:36] Speaker B: It's powerful stuff.
[00:05:38] Speaker A: But before we go too deep, can we take a step back and explain how this magic actually happens?
[00:05:43] Speaker B: Yeah.
[00:05:43] Speaker A: What's going on behind the scenes with these AI powered tools?
[00:05:47] Speaker B: Let's demystify it a little bit. It's all about causal analysis. We're trying to figure out, you know, what is actually causing a particular outcome, not just what's, you know, happening alongside it.
[00:05:58] Speaker A: Right. So we're talking about cause and effect.
[00:06:00] Speaker B: Cause and effect.
[00:06:00] Speaker A: Not just correlation.
[00:06:02] Speaker B: Not just things that happen together.
[00:06:03] Speaker A: Right.
[00:06:04] Speaker B: The things that truly drive an outcome.
[00:06:06] Speaker A: Okay, but how do we actually do that?
[00:06:08] Speaker B: Yeah.
[00:06:09] Speaker A: What are the tools and techniques that are involved here?
[00:06:11] Speaker B: Yeah, so there's a few different algorithms that kind of come into play, but the basic idea is we want to create that accurate virtual control group.
[00:06:20] Speaker A: Okay.
[00:06:20] Speaker B: One that truly mirrors the attributes of the group that received the marketing treatment.
[00:06:26] Speaker A: Right. So it's all about creating a more accurate comparison group. More accurate comparison that accounts for all those hidden factors. Hidden factors that could be skewing our results.
[00:06:36] Speaker B: That could be skewing the results. Exactly.
[00:06:38] Speaker A: Oh, I mean, this all sounds pretty complicated.
[00:06:40] Speaker B: It can sound a little complicated.
[00:06:42] Speaker A: Is this something that, you know, five, ten years ago only, like, giant companies with huge data science teams could do?
[00:06:49] Speaker B: Yeah. I mean, even a few years ago, you would have needed a team of data scientists using tools like R or Python.
[00:06:56] Speaker A: Right.
[00:06:57] Speaker B: To do this kind of analysis.
[00:06:59] Speaker A: But now with platforms like G2M Insights, we can do this without needing to write a single line of code.
[00:07:06] Speaker B: Without writing any code.
[00:07:07] Speaker A: Right.
[00:07:08] Speaker B: It's really democratizing access to these powerful tools.
[00:07:11] Speaker A: That's amazing.
[00:07:12] Speaker B: And importantly, you know, these algorithms have been rigorously tested. There was a benchmarking study that actually tested them against synthetic data sets where the true answer was known.
[00:07:24] Speaker A: So we're not just, you know, kind of throwing darts in the dark here.
[00:07:28] Speaker B: We're not throwing darts.
[00:07:29] Speaker A: We have evidence that these things actually work.
[00:07:31] Speaker B: We have evidence that they work.
[00:07:32] Speaker A: That's reassuring.
[00:07:33] Speaker B: Yeah, absolutely. And. And one interesting finding from that study was that propensity score matching.
[00:07:39] Speaker A: Okay.
[00:07:39] Speaker B: Which is the most common method, performed exceptionally well.
[00:07:43] Speaker A: Good to know. To know. So I want to shift gears a bit and talk about, you know, all this applies to your work.
[00:07:49] Speaker B: Yes.
[00:07:49] Speaker A: Sales and marketing executives.
[00:07:51] Speaker B: Yeah.
[00:07:51] Speaker A: Because at the end of the day, this isn't just about, you know, fancy algorithms.
[00:07:55] Speaker B: It's not about the algorithm.
[00:07:56] Speaker A: This is about making smarter decisions, getting better results. Better results for your business.
[00:08:01] Speaker B: Right.
[00:08:01] Speaker A: Couldn't agree more.
[00:08:02] Speaker B: So let's walk through, like, a hypothetical scenario.
[00:08:05] Speaker A: Okay.
[00:08:05] Speaker B: Imagine you are trying to understand the impact of a recent email campaign on customer conversions.
[00:08:14] Speaker A: Okay.
[00:08:15] Speaker B: So you send out two versions of the email.
[00:08:18] Speaker A: Two versions.
[00:08:19] Speaker B: One with a discount code, one without.
[00:08:21] Speaker A: Classic AB test.
[00:08:22] Speaker B: Classic a B test. Right. And then you track how many people in each group make a purchase.
[00:08:27] Speaker A: Yep.
[00:08:27] Speaker B: That's something we all do all the time.
[00:08:29] Speaker A: All the time.
[00:08:30] Speaker B: But I'm guessing there's more to it than just looking at the conversion rates of each group.
[00:08:34] Speaker A: There's always more to it.
[00:08:36] Speaker B: Right.
[00:08:36] Speaker A: You know, the naive approach would be to simply compare the conversion rates and say, okay, well, the discount code is a winner because it had a higher conversion rate.
[00:08:44] Speaker B: Right.
[00:08:45] Speaker A: But that doesn't tell us the whole.
[00:08:46] Speaker B: Story, because what if the people who received the discount code were already more likely to convert for other reasons?
[00:08:51] Speaker A: Exactly. Right. What if they were already loyal customers or had items in their shopping cart?
[00:08:56] Speaker B: You know, those factors could be influencing their decision to purchase regardless of the discount.
[00:09:01] Speaker A: Totally.
[00:09:02] Speaker B: So that's where the AI powered analysis comes in.
[00:09:04] Speaker A: Okay.
[00:09:05] Speaker B: Using propensity score matching or other similar techniques.
[00:09:07] Speaker A: Right.
[00:09:08] Speaker B: We can create that virtual control group that mirrors the characteristics of the group that received the discount code.
[00:09:15] Speaker A: So we're essentially creating a more accurate comparison.
[00:09:18] Speaker B: More accurate.
[00:09:19] Speaker A: By controlling for all those other variables.
[00:09:21] Speaker B: Other variables. Exactly.
[00:09:22] Speaker A: That could be influencing the outcome.
[00:09:24] Speaker B: That could be influencing the outcome.
[00:09:27] Speaker A: And this allows us to isolate the true impact. True impact of the discount code.
[00:09:31] Speaker B: Of that discount code. Yes.
[00:09:33] Speaker A: We might discover that the discount didn't actually drive as many conversions as we thought.
[00:09:37] Speaker B: Or even that it had a negative impact.
[00:09:39] Speaker A: Or even that it had a negative impact. Wow. Okay, that's. That's powerful stuff.
[00:09:44] Speaker B: Powerful stuff.
[00:09:44] Speaker A: That makes sense.
But how do we actually do this in practice?
[00:09:49] Speaker B: Yeah.
[00:09:49] Speaker A: What are the steps involved in setting up and running this kind of analysis? So let's break it down into, like, actionable steps. What does this process actually look like in G2M insights?
[00:10:01] Speaker B: So it's a. It's a five step process, really intuitive.
[00:10:04] Speaker A: Okay.
[00:10:04] Speaker B: The first step is creating a data set. So bringing together all that relevant information about your customers or prospects.
[00:10:11] Speaker A: Okay, so like demographics, purchase history, website activity, that kind of stuff.
[00:10:15] Speaker B: Exactly. All that good stuff.
[00:10:16] Speaker A: Okay.
[00:10:17] Speaker B: Then you're going to define the variables for your model.
[00:10:19] Speaker A: Okay.
[00:10:20] Speaker B: Including the specific outcome you're trying to measure. You know, is it conversions, is it churn?
[00:10:25] Speaker A: Right, right.
[00:10:26] Speaker B: The third step is all about exploring your data.
[00:10:28] Speaker A: Okay.
[00:10:28] Speaker B: You're making sure it's clean, it's complete, it's accurate.
[00:10:32] Speaker A: Data quality is key.
[00:10:33] Speaker B: Data quality is key.
[00:10:34] Speaker A: Data quality is key.
[00:10:35] Speaker B: It's garbage in, garbage out, as they say.
[00:10:37] Speaker A: Exactly.
[00:10:38] Speaker B: Okay. So we've created the data set, we've defined our variables, we've made sure the data is clean.
[00:10:43] Speaker A: Then you get to the fun part.
[00:10:44] Speaker B: Okay. This is where the magic happens.
[00:10:45] Speaker A: This is where the magic happens. You configure and train your model, choosing the right algorithm and letting the AI do its work.
[00:10:53] Speaker B: Okay.
[00:10:54] Speaker A: And then finally you interpret the results.
[00:10:56] Speaker B: Understanding what the analysis is telling you about the effectiveness of your marketing efforts.
[00:11:01] Speaker A: So we've gone from raw data to actionable insights.
[00:11:05] Speaker B: Actionable insights.
[00:11:07] Speaker A: That's the goal.
[00:11:07] Speaker B: That's what we want. Right.
[00:11:09] Speaker A: And I know this might sound a little daunting to some folks, but. Yeah, with a platform like G2M Insights. Yeah, you know, it's. It's really quite straightforward.
[00:11:17] Speaker B: It's very straightforward. The platform guides you through each step of the process. It provides helpful visualizations and explanations along the way so you're not just, you know, left to your own devices to figure it out.
[00:11:28] Speaker A: That's good to hear. That's good to hear.
[00:11:29] Speaker B: Yeah.
[00:11:29] Speaker A: Now, I know a lot of sales and marketing executives might be thinking, you know, this all sounds very technical.
[00:11:35] Speaker B: Yeah.
[00:11:36] Speaker A: Do I need to be a data scientist to use these tools?
[00:11:39] Speaker B: Not at all. Not at all. G2M Insights is designed to be user friendly and accessible, even for those without a strong technical background. You don't need to write any code. You don't need to understand complex algorithms.
The platform does the heavy lifting for you.
[00:11:55] Speaker A: So it's like having a data science team built right into the platform.
[00:11:58] Speaker B: It's like having a data science team in your back pocket.
[00:12:01] Speaker A: I like that. I like that. And that can be a game changer for businesses of all sizes.
[00:12:06] Speaker B: Oh, absolutely.
[00:12:07] Speaker A: Means you can unlock the power of this AI driven AB testing analysis without needing to invest in, you know, expensive resources or specialized expertise.
[00:12:18] Speaker B: Exactly.
[00:12:20] Speaker A: Okay, I'm sold. But I'm sure some folks are still wondering, like, what's the real world impact of all this?
[00:12:26] Speaker B: Yeah.
[00:12:26] Speaker A: How does this translate into tangible business results?
[00:12:30] Speaker B: You know, at the end of the day, it's about making more informed decisions about your marketing campaigns.
[00:12:35] Speaker A: Right.
[00:12:35] Speaker B: Having a clear understanding of what's truly driving results, what's not.
[00:12:39] Speaker A: Okay.
[00:12:39] Speaker B: And that allows you to optimize your campaigns for maximum roi.
[00:12:43] Speaker A: So no more wasted spend on campaigns that aren't delivering.
[00:12:45] Speaker B: No more wasted spend. No more shooting in the dark.
[00:12:48] Speaker A: Music to any marketer's ears.
[00:12:50] Speaker B: Music to their ears.
[00:12:51] Speaker A: And beyond that, you can use these insights to improve your targeting, personalize your messaging, and ultimately create more effective customer experience.
[00:13:00] Speaker B: Cover customer experiences. Absolutely.
[00:13:01] Speaker A: It's all about getting the right message to the right person at the right.
[00:13:04] Speaker B: Time, at the right time with the right message.
[00:13:06] Speaker A: And AI powered AB testing is an incredibly powerful tool for achieving that.
[00:13:11] Speaker B: Incredibly powerful.
[00:13:12] Speaker A: This has been an eye opening, deep dive into the world of AI powered AB testing.
[00:13:17] Speaker B: Yeah. It's a. It's a fascinating space that's constantly evolving.
[00:13:20] Speaker A: It is. Any final thoughts for our listeners as they embark on their own journey to marketing mastery?
[00:13:27] Speaker B: I'd say start experimenting. Don't be afraid to dive in. Try these tools out. I think you'll be really amazed at the insights you can uncover and the.
[00:13:35] Speaker A: Impact it can have on your business.
[00:13:36] Speaker B: The impact on your business. Exactly.
[00:13:38] Speaker A: That's great advice. And remember, G2M Insights is there to guide you every step of the way.
[00:13:42] Speaker B: Absolutely.
[00:13:43] Speaker A: So thanks for joining us today and happy analyzing.
[00:13:46] Speaker B: Happy analyzing.
In this episode of the Insights Achieved Podcast, we explore the art and science of A/B testing and how it can supercharge your marketing effectiveness. Inspired by insights from our G2M blog, we discuss how A/B testing empowers marketers to make data-driven decisions, optimize campaigns, and achieve better results with fewer resources. Discover practical tips for designing effective tests, interpreting results, and applying findings to improve performance across channels. Whether you’re a seasoned pro or testing for the first time, this episode offers actionable strategies to get more from your marketing efforts. This podcast episode was generated using AI […]
