Whenever you want to make updates to your Shopify store, even if it’s something as small as changing some font or adding a GIF, you would be wise to run an A/B test rather than just publishing the edits for everyone right away.
Indeed, A/B testing turns ecommerce into a science, allowing you to conduct rigorous experiments and objectively determine what it takes to reduce your bounce rate, increase your conversion rate, and achieve whatever other goals you may have for your store.
As one of the most commonly used tools for A/B testing on Shopify, VWO offers all the features you need to experiment with changes to your store, though it is a little pricey and doesn’t exactly have the most seamless integration with Shopify’s interface.
Below, we’ll cover everything that Shopify merchants need to know about using VWO, including:
Assuming you already have a VWO account, here are the steps that you would need to take to integrate VWO into a Shopify store:
Now that the VWO SmartCode has been installed, you need to enable VWO to track your Shopify events:
You’ll find that you have a few other options for setting up your VWO installation as well:
After you’ve installed the VWO SmartCode and enabled tracking on Shopify, you can now set up an A/B test through VWO:
Toward the left side of your screen, you’ll see that you can toggle between three tabs of settings for your newly created test: “Configuration”, “Review”, and “Reports”. Go to the “Pages” section of the “Configuration” tab and enter the URL of the page on your Shopify store that you want to test.
Next, go to the “Variations” section of the “Configuration” tab and select “Add variation”. You can use VWO’s visual editor or code editor to add whatever changes you want to the new variation.
Both the visual editor and code editor are helpful for those with web development experience. Whenever you select an element on the page in the visual editor, VWO details exactly where the element lands in the HTML hierarchy of the page.
Once you’re done making your changes to the variation, move on to the “Metrics” section of the “Configuration” tab.
You’ll need to set a primary metric for your test, and you can add secondary metrics that will be tracked as well.
Specifically, the types of metrics that you can track with VWO include:
You also have the option of targeting specific segments of visitors with your test, based on factors such as:
VWO allows you to build your own custom segments as well, so you can combine multiple conditions into the same segment if needed (e.g., targeting only mobile users who are in a certain location during a particular range of hours).
It’s also worth noting that VWO allows you to customize the traffic split for your test, meaning that instead of half your traffic going to the control and half going to the variation, you can set it up so that 55% goes to the control and 45% to the variation, 60% to the control and 40% to the variation, etc.
You should strongly consider taking advantage of this setting, as it’s a bit risky to send half of all your page traffic to a variation with changes. After all, your changes could end up having a negative impact on user experience.
By sending more traffic to the control than the variation, you’ll be able to minimize the damage if things go wrong with your test — just keep in mind that you may want to run your test a little longer if you do use an unbalanced traffic split, as your variation might then need more time to collect enough data to generate statistically significant results.
Once you’ve set up your metrics, segments, and traffic split, select “Start Test” to get your experiment rolling.
VWO will prompt you to review your test before it goes live. This is always a good idea, as you don’t want to waste time on a test that contains typos or any other mistakes — in these cases, you wouldn’t be able to tell whether a drop in performance was due to an accidental error or what you were truly trying to test. Furthermore, such mistakes are never good for your brand image, of course.
To see how your control and variation are performing, open the “Reports” tab on the settings page for your test.
For each metric that you decided to track when configuring your test, VWO breaks down the performance of each variant according to both the empirical data and more in-depth statistical analysis.
The empirical data is simply the raw numbers collected during your test, such as the total number of visitors and conversions. For binary metrics like add-to-cart rate, this column in the chart will be labeled “Unique Conversions/Visitors”. For real-valued metrics, such as revenue per visitor, it will be labeled “Total (Conversions/Visitors)”.
In some cases, the empirical data might be all that’s needed to determine whether or not the changes you made in the variation are having their intended effect. For example, maybe your control and variation have roughly the same number of visitors, but your variation has more than twice as many conversions — no more information would be needed for you to feel comfortable fully publishing your changes to the page.
But it’s usually not that simple. What if, according to the raw numbers, your control has a conversion rate of 15% and your variation has a conversion rate of 15.5%? Is that improvement significant enough for you to go ahead and publish your changes, or could it all just be due to random chance? And with a difference that small, your variant might actually perform worse than the original version of the page in the long run…
This is where the other columns in your VWO report come into play.
The “Expected Improvement” column will show you how much better or worse you can expect your variation to perform compared to the baseline. If you toggle the “In-depth data review” option on for your report, this data will be displayed as a box chart.
The last column in your report (labeled “Decision Probabilities” when the “In-depth data review” option is toggled on and “Probability of Better” otherwise) will tell you the odds of whether your variation is statistically better (or worse) than the control. Typically, a score of 95% or higher in the right direction indicates that you should make the full switch to your variation. With this analysis, it’s easier to determine whether the variation or the control is the real winner of the test, even when the raw numbers don’t look that different.
If you have determined that your variation is indeed more effective than the original version of the page, here’s how you can use VWO to implement your changes for good:
Once you’ve rolled out a variation, you will be able to access it by selecting the “Web Rollout” option in the left sidebar of the main VWO control panel.
The campaign dashboard here will allow you to see how many people have visited the page since you rolled the variation out. You can also pause, delete, or archive the web rollout campaign if you would like to restore the page back to its original design.
The Shogun A/B Testing app makes it especially easy to experiment with changes to your Shopify pages.
This is because Shogun A/B Testing is natively integrated within Shopify, allowing you to design test variants directly with Shopify’s built-in theme editor. You can set up your campaigns and evaluate the results without ever leaving your Shopify account as well. Compared to truly third-party platforms like VWO, there are far fewer steps that you’ll need to take to get your A/B tests up and running.
Another advantage to using Shogun A/B Testing is the price. VWO has a heavyweight feature set — and merchants should expect to pay up for those features. On the other hand, Shogun offers a lower price point, requiring less of a financial commitment. And even though the cost is low, Shogun still offers advanced capabilities, as you can target your test to a particular segment of your visitors and customize the traffic split between your control and your variant.
Overall, if you want an affordable A/B testing solution that’s easy to implement on Shopify, then you should certainly consider Shogun — you can even try out the app for free.
