Getting Started
The app offers 4 types of tests. On launching the app, you’ll be presented with the following screen in Shopify, where you can choose which type of test you wish to start.
If you have a draft or a finished test, you will be dropped to the dashboard instead and you would select your next test and type via the dropdown to the right.
1. Page Tests
Compare different versions of a specific page. Useful for testing:
Product page layouts
Collection page designs
Landing page variations
Blog article layouts
2. Template Tests
Evaluate multiple template versions across pages using the same template. Ideal for:
Testing changes across all product pages
Optimizing collection page templates
Improving blog article templates
3. Theme Tests
The Original theme in any theme test is always your shop’s Main theme.
Once a theme test is active:
First-time visitors are randomly assigned to either the Original or B variant.
This assignment is persistent — shoppers will continue to see the same theme variant across visits.
Analytics will reflect the assigned variant throughout the customer’s journey.
As a merchant, if you want to preview different theme variants:
Use an incognito window to view your live site.
Because assignments are random, you may need to open/close incognito windows multiple times to get the other variant.
Only one theme test can be run at a time
4. Split URL Test (new)
We have just released this new test type for a Split URL Test.
This enables an easy workflow to test between different Landing Pages on your store, or even when you are using a third-party landing page builder.
Theme Tests + Page/Template Tests or Personalization
Theme tests can run simultaneously with:
Page tests
Template tests
Personalization experiences
Here’s how they interact:
If a shopper isn’t locked into a theme yet, the theme test assignment happens first.
If they’re locked into the Original (Main theme):
They can also be served a variant of another test or a personalization.
If they’re locked into the B variant (not the Main theme):
They will not see any other test or personalization — those only apply to the Main theme.
Analytics will record:
The shopper’s theme variant
Any other test or personalization variant on that same theme (when the visitor meets the conditions)
Running Your Tests
Creating a Test
Select the test type (page, template, or theme)
Choose the target page or template
Create variant B (variant A is the original)
Configure goal metrics
(Optional) Set audience targeting
Review and launch the test
Test Status
Tests can be in one of three states:
Draft: Test setup in progress
Running: Test is actively collecting data
Finished: Test has been completed
Starting a Test
Configure test parameters
Review the setup
Click “Start Test”
When your rest is running, you can check the presentation of each variant via the ‘three dots’ in the dashboard.
Stopping a Test
Review results
End the test
Promote the winning variant
Navigating the Dashboard
The dashboard shows you:
Drafted, In Progress, and Finished Tests
Search Function for finding tests
Sort Function
From the dashboard, you can click on any test to reveal detailed analytics on in-progress and finished tests. In the case of a draft test, clicking the test in the pane will take you back to the setup screen if the draft was abandoned.
You can also delete drafts from the three dots or view your live varints for in progress tests.
Audiences
From here, you can define segments of customers you can specifically target for tests. You’ll find a prompt to ‘build an audience’ here.
Goal Metrics
Measure test success with varying metrics:
Conversion Rate: Percentage of visitors who complete a purchase (Orders / Total Sessions)
Clickthrough Rate: Percentage of visitors who click specific elements (Useful for testing CTA effectiveness)
Add to Cart Rate: Tracks how often visitors add products to their cart (Ideal for product page optimizations)
Please note that for the Add to Cart event, the Add to Carts are only tracked if there are add to cart buttons on the page that is being tested.
Form Submissions: Measures the completion rate of forms (Requires form ID tracking)
You'll require a Form ID when the goal is set to form submission. In order to get a form ID, you need to follow these steps -
Open the page with the form.
Inspect the page and the form in the browser.
Ctrl+Shift+C for Windows, or Command+Option+C for macOS.
Locate the form class while clicking on the form, and next to the "id" copy the ID.
Paste the form ID under the Form ID option of the Form Submission goal in the app.
Revenue Per Visitor: Tracks net sales averaged to visitor sessions (Total Sales / Total Sessions).
Average Order Value: Tracks average order value, factoring in discounts (Total Sales / Total Orders).
Custom Goals: Tracks your own custom events, designed for advanced use cases like tracking third party form submissions, click on a product image carousel button, or clicks on a specific button.
Custom Goals Set Up
Custom Goals requires custom Javascript to be added to your theme to trigger your custom event where desired. Let's walkthrough an example for creating a custom event for tracking a Typeform Form Submission.
Configure your test
Create a new test as usual. In the Test Goals section, select Custom event.
Enter the name of the event you'd like to track (e.g.,
typeform_submission).Copy the custom event code that we will use in step 2.
Fire the event on your theme
In your Shopify theme code, add a script that triggers the event that you copied from step 1. The format of the script will depend your theme and custom event. Feel free to contract our Support team or your developer if you need help.
To ensure that this script is available on all of the pages that are part of your test, add this script to the
theme.liquidfile right above</head>.Paste this code into your theme file to add the custom event:
<script>
window.addEventListener('message', function (event) {
if (event.data && event.data.type === 'form-submit') {
Shopify.analytics.publish('shogun:custom_event', {name: 'typeform_submission'});
}
});
</script>Note: this script assumes you have Typeform’s Embed SDK already on your storefront.
Publish test and view results
Go to your test's analytics page in the Shogun app.
Look for the Custom Event Rate graph. (May take a few minutes to update.)
This shows the percentage of sessions in which the event was triggered—calculated per variant.
Metrics Tracked
Each test tracks the following key metrics:
Page Views: Number of times a page is accessed.
Clickthroughs: Recorded when a user clicks on a link or button.
Add to Carts: Tracked when a visitor adds a product to their cart.
Form Submissions: Number of successfully submitted forms.
Conversion Rates: Percentage of users completing a desired action (e.g., purchase, sign-up).
Total Sales: Total revenue generated.
Revenue per Visitor: Average revenue per visitor.
Average Order Value (AOV): Average amount spent per order.
Audience Targeting
Customize tests for specific segments:
Device types
New vs. returning visitors
Geographic location
Logged-in status
Traffic source
URL patterns
Results Analysis
Metrics Dashboard
Real-time performance tracking for each variant
Visual graphs showing trends over time
Detailed conversion metrics
Traffic distribution insights
Statistical Significance
Confidence threshold: 95% (confidence threshold will be displayed when the value is >35%)
Variant Performance Details
Each variant includes:
Key metrics overview
Improvement over control
Traffic split percentage
Preview links
Test Concurrency & Traffic Distribution
The app enables multiple tests to run simultaneously using a Shopify theme-layer traffic distribution system. This ensures:
Fair traffic allocation
No test interference
Consistent user experiences
Accurate metric tracking
Traffic Distribution Methods
Even Distribution (
even)Uses session IDs for consistent assignments
Evenly divides traffic across tests
Uses a 10,000-bin system for fairness
Greedy Distribution (
greedy)Default method
Assigns users randomly to variants
Uses local storage for consistency
Supports dynamic traffic split adjustments
Personalization & A/B Test Interaction
Personalization rules are evaluated first.
If a personalization rule applies, it may:
Override an A/B test (if the test variant is the control)
Allow the A/B test to proceed (if using a non-control variant)
This ensures personalized experiences do not disrupt test validity.
Implementation Details
Visitor Assignment
Checks for existing variant assignment in local storage
Evaluates audience targeting rules
Assigns users based on the chosen distribution method
Caches assignments for consistency
Template Loading
Loads the correct template variant
Supports theme preview mode
Manages theme redirections when needed
Maintains preview bar functionality
Concurrent Test Management
Tests are stored in Shopify metafields
Each test includes:
Target (page/template/theme)
Audience rules
Variant configurations
Traffic split percentages
The system dynamically adjusts traffic distribution for multiple tests
Cache Management
Variant assignments are cached in local storage.
Cache can be bypassed with
?cache=falseURL parameter.Optimizations refresh every 24 hours.
Audience Targeting in Concurrent Tests
Audience targeting rules are evaluated independently for each test.
Targeting parameters include:
Device type
Visitor status (new/returning)
Geographic location
Login status
URL patterns
Referrer sources
Best Practices for Concurrent Testing
Test Isolation: Avoid overlapping tests on the same elements.
Traffic Management: Ensure adequate traffic to achieve statistical significance.
Duration Planning: Consider longer durations for split traffic.
Monitoring & Analysis: Regularly review test interactions and outcomes.
Best Practices for using the Tool
Test Duration
Run tests until statistical significance is met.
Monitor conversion rates and raw numbers.
Traffic Split
Default split: 50/50.
Adjust based on traffic volume and risk tolerance.
Goal Selection
Align goals with business objectives.
Use multiple metrics for comprehensive analysis.
Prioritize metrics impacting revenue.
Variant Design
Focus on meaningful changes.
Test one significant change at a time.
Document variant differences.
Technical Details
Implemented through Shopify themes and templates.
Real-time metric processing.
Automatic variant screenshots.
Seamless Shopify admin integration.
Supports multiple concurrent tests.
FAQ
How does Shogun avoid the "Flash of Content" problem?
Shogun prevents a flash of content by intercepting the page request before any content is rendered by the browser, redirecting to the appropriate variant using Shopify view parameters. Rather than manipulating the DOM after load, this server-side routing ensures the correct content is delivered immediately. After rendering, Shogun updates the browser’s address bar to reflect the original URL the user navigated to.
Does Shogun impact my page performance?
With all rendering is done by the Shopify platform, Shogun ensures we have all data available to make a decision on what variant to render within the initial page load and interrupt the process as early as possible to prevent any unnecessary delays. There will be a very minimal performance impact because the user's browser starts to load the initial request and then needs to make another request to get the selected variant.
Our performance impact is greatly reduced, compared to other testing platform, by not having to make external calls to determine variant selection, which subsequently requires swapping content on the page which leads to the flash of content problem.