The End of Interesting: AI in Experimentation with Deborah O’Malley [Podcast]

Conversion Sciences

In this episode of Intended Consequences, Brian Massey sits down with Deborah O’Malley, founder of GuessTheTest.com, to explore the fast-changing world of AI in experimentation — from A/B testing myths to the ways AI is already changing how digital marketers approach conversion optimization. And yes, they really do debate whether AI will kill our creativity. 🎧 Listen to the full episode here: Ready to improve your site’s performance? Learn more about Conversion Science’s Optimization Services. Are You Running the Wrong A/B Tests? Deborah pulls no punches: “People are still testing button colors and headlines when they should be testing concepts.” That’s one of the many interesting misconceptions about experimentation. But as she explains, the most valuable insights come from big, bold tests — especially ones that challenge your brand’s assumptions. And with the arrival of generative AI, we now have the ability to scale those tests like never before. How AI in Experimentation Is Changing the Game AI isn’t just writing headlines — it’s redesigning the entire optimization process. Deborah and Brian explore a few critical shifts: 1. Faster Hypothesis Generation AI can instantly produce dozens of test ideas based on your existing content, analytics, or competitor sites. This helps marketers and CRO pros move from analysis paralysis to active testing — fast. “You can use AI to brainstorm variations you’d never think of on your own,” Deborah explains. “That means more creative testing… not less.” 2. Pattern Recognition at Scale While most human optimizers rely on gut instinct or anecdotal evidence, AI can spot trends in user behavior across massive datasets. That means smarter test prioritization, better personalization, and tighter feedback loops. 3. Test Ideas From Outside Your Echo Chamber AI doesn’t share your brand biases — and that’s a good thing. By using tools like ChatGPT to simulate how different personas react to your copy or design, you can explore radically new angles without waiting for an actual test to finish. From “Interesting” to “Impactful” One of Deborah’s boldest claims: We should stop chasing ‘interesting’ test results. Why? Because what’s interesting isn’t always what moves the needle. “You want tests that are valid, repeatable, and drive real business results. That’s where AI can help — it brings a level of objectivity and scale that humans alone can’t match.” In other words, AI in experimentation isn’t replacing us — it’s upgrading us. Want to Test Better? Start Here. Whether you’re a CRO veteran or just getting started with testing, this episode is packed with practical insights: ✅ Which A/B tests are still worth running✅ How to think beyond copy tweaks and start testing experiences✅ Why generative AI might be your best brainstorming partner✅ And how to avoid common pitfalls that make test results meaningless And don’t worry — it’s not all tech talk. Deborah and Brian also cover the human side of experimentation, from internal politics to the fear of failure. Learn more about conversion-focused web design and redesign that achieves better results faster. Final Thoughts: Embrace AI in Your Testing Workflow This episode challenges us to move past the superficial and start building testing programs that matter. Whether you're optimizing landing pages, ecommerce funnels, or entire customer journeys, AI in experimentation is the lever that can help you scale faster and learn deeper — without sacrificing creativity. “When used well, AI becomes the co-pilot of every test you run,” Brian says. “It accelerates creativity, supports analysis, and helps you ask better questions — the real key to CRO.” Want to improve your testing strategy? 💡 Talk to a Conversion Scientist and start testing what actually matters. 🔬 Learn about our fully-managed Conversion Optimization Services Links