How Do I Perform an Email Split Test?

One of the best ways to drive engagement with your email content is by using split tests, or A/B tests, to determine is specific changes to your content or layout increase response. When testing, you'll send out two versions (A and B) of your email. Version A, also called your control email, will have the content you might typically send, and then you'll create an alternate, Version B, in which you make a change, either to the layout, the call to action, graphics, subject line, or whatever else you'd like to test. It's best to focus on making just one change, as this will help determine if that particular change drives more engagement.

To conduct your test, you'll send both versions to a small, randomized subset of your overall email audience, measure the response from each group, and then deliver whichever version performs best to to the rest of your email audience.

To do this in Virtuous, hover over the left side bar, then select Marketing. Under Marketing, select Email.

Then, create a new email, either from a template or by clicking on the Actions button in the upper right to create a new email from scratch.

On the Email setup screen, you'll see a toggle option to "Send A/B Split Test." Click to begin configuring a split test.

This will open the split test setup screen.

At the top of the screen, you'll see the name and basic information for your email, which will now be the name of your split test. Below, you'll see a slider to determine how you'd like to break up your email send. By default, split test will send 20% of your audience the test of version A, 20% the test of version B, and then the winning version will be sent to the remaining 60% of your recipients. Just grab and drag the slider to adjust these percentages.

You can adjust the size of each test audience to anywhere from 5% to 25% of the total recipient group. The size of the test groups will always be identical.

Below the slider, you'll see two options for determining your winner: Open Rate (the default option) and Click Rate. Use Open rate if you are testing two different subject lines, or two different custom senders for your email. If you'd rather see which email inspires more recipients to take action, use Click Rate instead. This is a great way to test two different calls to action, layout changes, and other email content.

Use the Test Duration fields to determine how long you'd like to wait (between 2-72 hours) to see results from your test before determining a winner and sending the winning version. As a general rule, the longer you can wait, the better your results will be, as you should have more data informing the outcome.

Next, you'll enter information for each version of your split test.

Be sure to give each version a unique name, so they are easy to identify. You can choose to use the same subject line and sender for each, or enter different values if that's what you are testing.

Once you have entered all of the information for your test. click "Next" in the bottom right to create or edit your email.

Once you begin editing, you will be using the same email editor that is used for creating regular bulk or automation emails. You can use the same steps outline in the "Build" section of this article to configure your email, just the way you'd like.

One difference is that, in the upper-right section of the screen, you'll see a button to toggle between versions A and B of your test.

Whichever button is brighter will tell you which Version you are viewing. In the above example, you would be on Version A. Click on "Version B" to switch to that version instead. Begin by creating or editing Version A (which would be your "control" version) and then toggle to Version B. You'll see that Version B initially is a copy of A. Then, you can make edits to Version B (if desired).

At any point, you can click on the Actions drop-down in the upper right and select "Sync A/B Versions" if you'd like to make both versions identical again. This can be especially handy if you make significant changes to one version that you'd like to copy to the other.

You will have the choice to either sync Version A's content to Version B, or Version B's content to Version A.

You'll have the ability to preview your email, or even send a test email, just as you would with any other email. The preview or test will use whichever email version you are actively viewing.

Once you have saved and published your email, you can click "Next" to move on to target your recipient group and then schedule or send your email, just as you would with any email. On the final step, you'll see a count of recipients for each test version as well as for the winning version of your email.

When sending your test, keep in mind that a larger audience will always provide more actionable data. A split test must be sent to more than 100 recipients in order to have a large enough statistical sampling to determine a winner.

In any case when a winner can not be determined, either because of a small sample size or because the difference in results between versions is not large enough to determine a clear winner, then Version A will be declared the winning version by default.

Once your split test has been sent, you'll be able to view the results of your test by selecting it from the "A/B Test Results" tab of the Email screen.