Test & Learn

De-Mystifying A/B Testing On The Web

May 11, 2019

A/B testing is extremely common today—it’s rare if a growth team isn’t using some sort of testing somewhere in their growth efforts. Most often, though, we see teams leveraging A/B testing features that are already a part of SaaS tools they’re already using.

Tools like Intercom, Mixpanel and all sorts of CMS platforms have tools baked in that make it easy to A/B test messaging, content, etc., which is awesome—it’s a really exciting time to be in growth because tooling has democratized the ability to do previously complex things like testing.

If testing isn’t a built-in feature, though, things can start to get tricky. We’ve worked with people who are incredible at running all sorts of testing, even at scale, but if they had to test a button color from scratch, they couldn’t. It’s not that they don’t know how the process should work, it’s that the complexity of the process isn’t as ‘hidden.’

The good news is that you don’t need a fancy or expensive program to get started with A/B testing—and you don’t need to have programming skills. Google has made it relatively simple and painless to ramp up your testing and optimization efforts with free, relatively easy-to-use software. In fact, we’ve used the suite of tools we talk about in this post for all sorts of clients, ranging from startup to the enterprise.

Let’s demystify A/B testing and walk through an incredibly practical example that you can easily replicate.

First, The Tools

Google Analytics Account

The first thing you need to get started kicking off your A/B testing program is to have Google Analytics running on your website.

Don’t have it this already? Not a problem. Check out this guide on how to get your analytics code set up.

If you already have this, excellent. First step done.

Google Optimize Account

Google Optimize is Google’s solution for A/B testing and overall site personalization. It has an amazing WYSIWYG editor for a novice to handle along with custom JavaScript options for more advanced testing triggers and needs. Best of all, it’s free!

Next, let’s set up Google Optimize. Here is a tutorial from Google on the best practices for setting up your new A/B testing environment.

Important note: You must have a Google Analytics account prior to setting up Google Optimize.

Okay, we have a Google Analytics and Optimize account. Excellent. Let’s start testing.

Create Your First Test!

We look for small changes that yield big results. What can you add or remove to a page to determine if that increases your conversions or not? Need help brainstorming what to test, we are here to help.

Setting Up A Basic Test

For this test, we will test the difference between a white button and a yellow button on our site. We recommend testing individual, incremental changes, rather than testing radical variations.

1. Open Optimize.Google.com and click your website title.
Google Optimize list of accounts

2. Click Create Experiment.
Google Optimize list of accounts

3. Type in the name of your test. Learn more about the different test variations by following the links below. For this test, we will choose A/B, to test the difference between our original white button and variation of the yellow button.
  • A/B test: Tests two or more variants of a page. Also called an A/B/n test. Learn more
  • Multivariate test: Tests variants with two or more different sections. Learn more
  • Redirect test: Tests separate web pages identified by different URLs or paths. Learn more
  • Personalization: Personalize your page for targeted visitors. Learn more
Google Optimize list of accounts

4. Follow the step by step directions, outlined at the top of the screen:
Google Optimize list of accounts

5. Create Variants: Your original version will automatically be created as the “Original” Variant. Name the new variant in a way that outlines what will change between the original and the new version. You can add additional variants by clicking “Add Variant.”
Google Optimize list of accounts

6. The weight will automatically evenly split between the two variants 50% to 50%. Adjust the weight of targeting by clicking the weight number. To edit the weight of percentages, select “Custom Percentages,” instead of “Evenly Split.” You can then adjust the weight between your variants.

7. Add a rule to targeting. Learn More About Targeting Rules. You can choose to target among a range of parameters, including specific URLs, behavior on the site, query parameters, etc. For this test, we will target a specific URL. Add your targeting rules, based on the needs of your test.


Google Optimize list of accounts

8. Add a description to outline what you are changing and why. You may include your hypothesis or other pertinent information.
Google Optimize list of accounts

9. Link the account to Google Analytics, in order to measure your results. Click “Link to Analytics” and select the view.
Google Optimize list of accounts

10. Add an Objective. This is the website functionality you wish to optimize. You may choose from a list or create a custom objective. Learn More About Objectives. As your test runs, Google will report back on objective results.
Google Optimize list of accounts

11. In Settings, there are a range of other settings you may select. For instance, you may choose to target a certain percentage of your traffic (your targeting %s will then be divided among this portion of traffic), you may choose to set an activation event or receive email notifications.
Google Optimize list of accounts


12. For this test, our activation event is Page Load. Learn More About Activation Events Here

13. Now that we’ve set up the test, it’s time to update Variant 1. Return to the top of the page and select “Edit” for the Variant.
Google Optimize list of accounts

14. In Edit Mode, you will see menus for editing your page. Notice in the top right corner that Google tracks your changes. The menu at the top shows where in the code the element you are editing is located: “body > div > div > section > div >” For this test, we will change the button color of “What We Do,” found above the fold.
Google Optimize list of accounts

15. Clicking the button will update the options available to you in the pop-up “Edit Element” window. You can adjust the button’s destination, dimensions, typography, background, border and layout.
Google Optimize list of accounts

16. We adjusted our options to change the button from Original (white) to Variant 1 (yellow). In the Typography menu, we changed the text color from blue to white. In the Background menu, we changed the color from white to yellow.
Google Optimize list of accounts
Google Optimize list of accounts

17. When you’ve completed your changes, click “Save” and then “Done” in the top right corner.

18. To be sure that your changes went through and look good on desktop and mobile, select “Preview” from the main Testing screen.
Google Optimize list of accounts

19. Preview your changes on web, tablet, and mobile. You may also share the preview with others. If sharing with others, we recommend opening the sharing link in an incognito window.

20. When your test is set up correctly, previewed and functioning, click “Start.”
Google Optimize list of accounts

21. While your experiment runs, you can always view the results by selecting the “Reporting” tab, next to the “Details” tab.
Google Optimize list of accounts

22. Now that your test is live, monitor the results regularly! Google will gather data about the variants and find a winner between the two variants. We recommend running each test for 2 weeks.

23. When Google does find a clear leader, the Reporting dashboard will be updated. This dashboard will be available after the test is ended.
Google Optimize list of accounts

24. When you’re ready to end your experiment, simply click End from the Details Dashboard.
Google Optimize list of accounts
Subscribe to The Review and become a better growth leader.