Building a Culture of Optimization, Part 5: So You’ve Found a Big Win… Now What?

In part 5 of this 5-part blog series about ‘Building a Culture of Optimization’ I’m going to talk about the importance of sharing your wins and bringing your organization along with you. You can see part 1, part 2, part 3 & part 4 here. Part 5: So you’ve found a big win. Now what? Ensure you’ve double triple checked your results! Are they statistically confident? Did you control for external variables? Why is this important? A personal example… I ran a test where we found significant uplift over our control from a couple of test variations, but one version stood out as the clear winner. After closing the test, reviewing and analyzing the data, I communicated the results and recommendation to launch the winner to the rest of my organization. Most people
Continue reading...

Building a Culture of Optimization, Part 4: Evangelize the Process

In part 4 of this 5-part blog series about ‘Building a Culture of Optimization’ I’m going to talk about evangelizing your process within your organization. You can see part 1, part 2, & part 3 here. Part 4: Evangelize the process Process is important. Process leads to consistency, repeatability, and authority in a testing program. Sharing that process and getting others in your organization bought in and supportive is even more important. One source of truth One of the best ways to make your optimization program better known within your organization is to evangelize it via a widely accessible & visible roadmap. Here’s an example roadmap that I use within my organization: I host this roadmap in a Google doc that is accessible to everyone in my organization, from analysts
Continue reading...

Building a Culture of Optimization, Part 3: Know the Math

In part 3 of this 5-part blog series about ‘Building a Culture of Optimization’ I’m going to talk about the importance of bringing your organization up to speed on the math behind the tests. You can look back and see part 1 on the basics and part 2 on good test design. Part 3: Know the Math! Give your peers a short stats lesson (but keep it light)! What does someone running an A/B or MVT test need to know about math? How detailed should they be? Here is what I tell all of my coworkers: Statistical Confidence = confidence in a repeated result The confidence level, or statistical significance indicate how likely it is that a test experience’s success was not due to chance. A higher confidence indicates that: – the
Continue reading...

Building a Culture of Optimization, Part 2: Good Test Designs

In part 2 of this 5-part blog series about ‘Building a Culture of Optimization’ I’m going to talk about the importance of teaching good test designs. You can see part 1 about educating the basics here. Part 2: Good Test Designs You can’t have a good test without a good test design. One of the first things I do when a new test idea surfaces is sit down with the key stakeholders & test proposers to understand the details of what they’d like to test. We’ll talk through the variables that are going to be tested, how best to setup & design the test, and ensure we are on the same page in terms of potential test outcomes and how to ensure we are testing in a clean and consistent manner.
Continue reading...

Building a Culture of Optimization, Part 1 – Educating the Basics

I’ve presented at several conferences, webinars, and meetings recently on the topic of ‘Building a Culture of Optimization.’ It’s a great topic and something our industry as a whole is still in the process of nailing down – most organizations I see haven’t yet reached maturity here. Given the importance of building such a culture in order to drive testing and optimization across our businesses I’m writing a mini-series of posts on some of the most important things I see, do, and recommend. This will be a 5-part blog series on ‘Building a Culture of Optimization’ with parts 1-3 focusing on education, part 4 on process, and part 5 on spreading the word and shifting the culture. Part 1: The Basics – Educate educate educate! Educating your teammates is the
Continue reading...