How to Start AB Testing with Shopify in Just 7 Steps
What works for one eCommerce store owner likely won’t work for another. Even if you knew exactly what a competitor was doing to make a specific product move, you likely wouldn’t get the same results if you replicated everything about their marketing campaign and presentation.
Nothing works the same all the time when it comes to eCommerce.
When you want more purchases and more revenue from the traffic you receive, you have to figure out what works for your specific site, the products, the buyer’s journey, the customer, etc.
This is where Shopify A/B testing and multivariate testing come into play.
What is A/B Testing
A/B testing, also known as split testing, is a method of presenting visitors with two different versions of the same page. Some traffic will be served version A while others are served version B. Over the course of testing, you can use data to determine which version performs better or provides higher conversions.
This way, you can make data-driven decisions to continually improve the performance and growth of your Shopify store.
Without that data, you’re just guessing at changes and hoping for the best.
A/B testing in Shopify is simple in concept; 50% of the visitors to a certain page will see version A while the other 50% will see version B.
Version A would be the original version of a page, known as your “control”. Version B is your “variant”. If the variant outperforms your control, then version B becomes your new control and you create another variant in order to continue the process of improving results.
The ultimate goal of A/B testing is to take the traffic you already have and make it more valuable.
Why A/B Testing Matters in eCommerce
A/B testing in Shopify is critical for discovering what drives action for your target audiences. The changes made with each variant are often small in order to determine what change had the positive or negative impact.
Through A/B testing you can uncover changes to copy, images, CTA or testimonial placement that will help you build better ecommerce landing pages.
Beyond the improvements to specific product conversion and checkouts, you may uncover insights that can help throughout the buyer’s journey. For example, uncovering specific copy that consistently lifts conversions could help you rephrase your value proposition and reduce bounce rates on key landing pages.
The main benefits of using a Shopify split testing app include:
• Improved content engagement
• Reduced bounce rates
• Lifts in conversion rates
• Cart abandonment reduction
• Increased subscriptions/opt-ins
The following case study is just one of countless examples you can find online detailing the effectiveness of A/B testing.
Wall Monkeys, an online retailer of diverse wall decals, wanted to improve clicks and conversions on its home page.
In an effort to improve traction of the CTA on its home page, Wall Monkeys ran an A/B test comparing the original stock photo with something more whimsical that was a better fit for the brand.
That small change increased conversion rates by 27%
The next variant replaced the homepage slider with a prominent search bar which made conversions soar by 550%.
Keep in mind those kinds of results aren’t guaranteed. A/B testing is an experiment, so results can and will (always) vary. Your own results will be based entirely on statistical analysis and what you do with the data you uncover from one variant to the next.
The ability to analyze that data is the real benefit.
Shopify A/B testing takes a lot of the guess work out of growth and improvement while also greatly reducing the amount of time typically spent between comparing the performance of changes to your store.
The Difference Between A/B Testing and Multivariate Testing
While A/B testing deals with comparing the results of two versions (the control and the variant), multivariate testing allows you to compare more than one variant against your control.
So, instead of splitting your traffic in half between version A and version B, you might send 20% of your traffic to version A (your control) and divide the remaining 80% of traffic equally to versions B, C, D, and E.
The benefit of multivariate testing is that you’re not just testing multiple variants you’re also testing more than one element. This way you can see if elements combined in certain ways perform better than others, like a large CTA button with large images compared to a large CTA button with small images, small button with small images, small button with large images, and so on.
Note that you need a LOT of consistent traffic in order to run multivariate tests that generate useful data. If you’re not generating hundreds of thousands of unique visitors for your pages, it’s best to start with A/B tests in Shopify.
Which leads us to the next point…
Who Should A/B Test in Shopify?
If you’re curious whether or not you’ve got enough traffic, start by using a sample size calculator.
A sample size calculator allows you to input your current (baseline) conversion rate and then a minimum detectable effect. The minimum detectable effect is how much you want to increase your conversion by.
For example, with a baseline conversion of 3% and a 25% MDE, the calculator shows a minimum sample size of 8,408 visitors per variation. With a standard A/B test, that product page would need at least 16,800 visitors.
Not every Shopify store owner needs to run A/B tests. For new stores and those in the early growth phase, you’ll likely see better results by soliciting feedback directly from your customer base through surveys or by conducting user testing.
However, as traffic consistently grows you should incorporate A/B testing into pages consistently garnering the most traffic where you want to improve an action taken by your target audience.
Why Timing is Important with Shopify A/B Tests
The length of time you run Your Shopify A/B tests will vary, but ideally you should run the test for at least two full business cycles.
This is important because:
1. Customer shopping habits can change based on the day of the week
2. Certain customer segments may only shop on specific days
3. External factors like weather, seasons, events, and holidays can significantly impact sales
4. Customers may window shop and return later
5. You can account for various campaigns and anomalies like your newsletter dropping
Run your tests long enough to show statistical significance, rather than just stopping the test when you reach X number of conversions. This way you can reach your predetermined sample size with enough time to prove whether or not your changes are effective.
If you stop too soon, let the test run too long, or schedule during something like a major holiday shopping season then the validity of your test goes out the window.
While you can’t fully eliminate these external factors that impact the validity of your tests, you can minimize their impact by running tests for full week periods through multiple business cycles.
How to Start A/B Testing in Shopify
Before jumping into the how-to it’s important to mention that there’s some challenge with Shopify A/B testing.
The main issue is that the majority of Shopify merchants are on one of a few standard plans rather than Shopify Plus. Standard users aren’t able to modify the checkout funnel and instead use the checkout.shopify.com domain.
Because there’s cross-domain traffic during checkout, cookies are necessary for Shopify A/B testing tools to keep track of your users.
Unfortunately, some users block cookies. Likewise, some browsers (like Safari) may be set to block cookies by default.
If cookies are blocked, you automatically exclude a portion of users from your split testing and that can skew your results significantly if you’re factoring conversions and revenue into your tests.
For Shopify Plus users, it gets quite a bit easier.
Joshua Uebergang, author of Shopify Conversion Rate Optimization, provides one easy option if you’re familiar with coding:
“It is technically simple to run split-tests of visual elements on Shopify Plus stores. If you want to test “Add To Bag” text on the button of your product page against “Add To Cart”, you create a jQuery that selects a unique CSS identifier inside your favorite split-testing tool.”
If you’re not comfortable coding, the Shogun Page Builder for Shopify has built-in A/B testing that’s super easy to configure and manage.
How to Use Shogun to Setup Shopify A/B Testing
Step 1: Open Shogun
From the Shopify dashboard, click on Apps in the left nav menu, then click on the Shogun Page Builder to launch the app.
Step 2: Choose a page
Once the Shogun app launches, choose a page for your Shopify A/B test. Once you’ve chosen, click the “…” button for that landing page and choose “start AB test” in the dropdown.
Step 3: Choose your objective
Once you click “start AB test”, Shogun will launch the editor and automatically create a duplicate variant for you. To get started, choose an objective for your test.
Choosing “sales” or “add to cart” will immediately take you into the editor to begin adjusting your variant(s). If you choose “clickthroughs”, you’ll have an option to enter a destination for tracking those clicks.
Step 4: Make your changes
Within the editor you can make a variety of changes to compare how your variant(s) perform against your control. Remember to minimize the number of changes you make. Multiple changes in a variant make it difficult to determine which change had the most impact.
Step 5: Set the test options
When you’ve finished creating your variant, it’s time to configure the test. Click on “Publish” to bring up your testing options.
With Auto pilot on, Shogun will automatically publish the determined winner of the Shopify A/B test once it has confidence in a specific variant. If you turn Auto pilot off, you can specify the length of time you want the test to run.
You can also customize the percentage of traffic that will be directed to this page. Since most Shopify A/B tests involve a control and a single variant, the default is set to 50%, so traffic is split evenly between the pages in your experiment.
Step 6: Start the test
With options set, it’s time to launch. Click the button to start your test! Shogun will automatically begin randomizing the pages shown to visitors. Keep in mind while you’re running the test you won’t be able to make any changes. You’ll have to stop the test in order to make content updates.
Step 7: Review the data
When a test is finished, or when it’s stopped early, Shogun will display the results of the test including side by side comparison of the pages involved and data relating to your established objective. Use this data to decide the winner, create a new control, and setup a new Shopify A/B test.
Split test in Shopify as you scale
Not all merchants will have enough consistent traffic to in order to perform A/B tests, but those that do should always have tests running in order to establish and exceed objectives. With Shopify split testing, you can dramatically increase your store’s growth rate while reducing bounces and lost conversions. Rather than dumping budget into traffic acquisition, you can use Shogun to easily make changes in your store and build revenue with the traffic you already have.
Want to learn more about A/B testing in Shopify and eCommerce? Check out this detailed guide from Peep Laja of ConversionXL: How to Run A/B Tests