For the implementation of an emailing strategy and as in any marketing approach, it is preferable to perform conversion rate optimization tests (CRO for Conversion Rate Optimization).
> Download this email marketing optimization checklist” align=”middle”/>This practice is interesting insofar as the behavior of each audience is different. Something that works for one business may not necessarily work for another.
Optimization testing, however, can be complex, and without a minimum of attention it is possible to make decisions that could harm other parts of your strategy.
A / B testing is one of the easiest and most common optimization tests. The principle is very simple: it involves testing a variable of a marketing content against another, a green call-to-action and a red one, for example, to see which of the two is the most efficient.
This article provides a detailed analysis of how to set up an A / B test, as well as a complete checklist of the steps a marketer should follow before, during and after the test. You can bookmark this page for future reference.
Emailing: 7 steps to set up A / B testing
- Define the problem and areas for improvement.
- Set the objective to be achieved and the variable to be tested.
- Create two versions of emailing.
- Structure two segmented audiences.
- Define a statistical threshold for results.
- Define the duration of the test.
- Rely on user feedback.
To perform an A / B test, you must create two different versions of the same content by modifying only one variable. You will then show these two versions to two audiences of the same size and analyze which one offers the best performance.
For example, if you want to know if your homepage conversion rate increases when you move one of the CTAs from the side menu to the top of the page, create an alternate webpage that reflects the new CTA placement. The existing version, or control version, constitutes variation A, while variation B represents the alternative version.
To test these two variations, you will then present them to a predefined percentage of visitors.
To generate representative results that can inform your decisions, you need to carefully plan your test.
Define the problem and areas for improvement
The A / B testing process itself is based on a problem, usually identified through data analysis. Then, recommendations or potential improvement points are defined to set up the comparison with the original version.
After the identification phase, the problem must be described so that it can be understood by all those who will be involved in the A / B test. It is important to add details on the nature of the problem:
- Its perimeter.
- Its impact on strategy.
- Its level of importance.
Set the objective to be achieved and the variable to be tested
To assess the effectiveness of a change, you need to isolate a single variable and measure its performance. If you choose more than one, you will not be able to determine which one is affecting performance. You can test more than one variable in a page or in an email, but you should make sure that you test only one at a time.
Study the different elements of your marketing resources and their possible alternatives in terms of design, wording and presentation. Email subjects, sender names, and various options for personalizing your emails can also be tested.
Keep in mind that even the simplest edits, such as an image in an email or the wording of a call-to-action, can lead to big improvements. In addition, the impact of these kinds of changes is often easier to measure than the impact of larger changes.
Note: it is sometimes more relevant to test several variables rather than just one in a multivariate test.
To reinforce the relevance of the A / B test, it is advisable to focus on one main indicator, even if you subsequently wish to rely on several indicators per test. Set your primary indicator before setting up the second variant of the test.
Create two versions of emailing
The unmodified version of the item you are testing is the control version. If you choose to test your email marketing landing page, you should keep the layout and content that you normally use.
Once you have defined the control version, you can create a variant or an alternative, namely the marketing email that you want to compare to your control version.
For example, if you want to know if embedding a customer testimonial on a landing page would bring different results, set up your checkout page without a testimonial. Then create your variant with a testimonial.
The timing of your test plays a huge role in the results of your marketing campaign. If you were to test version A for one month and version B the following month, you would be unable to determine what factor is affecting the performance of the resource under test.
When performing A / B tests, you must test both variants simultaneously, otherwise your results will be unreliable.
Testing the timing itself, for example to determine the best time to send an email, is the only exception to this rule. This variable is particularly interesting, because depending on what your company offers and your subscribers, the optimal time to elicit their engagement can vary significantly from one industry and one target market to the next. other.
Structure two segmented audiences
Email testing should be done with two or more equal audiences to get conclusive results.
The method employed will depend on the A / B testing tool used. If you have access to the Enterprise version of HubSpot Marketing and do an A / B test on an email, for example, the traffic will be split automatically between the two variations so that they are served to a random sample. visitors.
The method you use to determine this factor will also depend on your A / B testing tool and the type of A / B testing performed.
On an email, it’s best to A / B test a small portion of your list to get statistically significant data. You can then choose the variant that generates the best results and send it to the rest of the list.
HubSpot’s emailing solution lets you determine your sample size using a slider. You can distribute your test to a sample split into two equal audiences, regardless of its size. However, it must have at least 1000 recipients if you choose a different distribution.
If you’re testing something that doesn’t have a defined audience, such as a web page, the length of the test has a direct impact on your sample size. You will need to run the test long enough to get a significant number of views, otherwise you cannot be sure of the difference in results between the two variants.
Define a statistical threshold for results
Once you have chosen your indicator, the definition of a statistical threshold makes it possible to justify the choice of the selected variant.
Concretely, the higher the percentage of your confidence level, the more you can be sure of your results. In most cases, a confidence level of at least 95%, or even 98%, is expected, especially if the setup process was time consuming. However, you can also use a lower confidence index if you are doing less precise tests.
The statistical threshold is sometimes compared to a bet where it comes down to deciding how much stake you want to place. Having 80% confidence that the winning version has the best design and betting everything on it is like performing an A / B test at an 80% threshold and picking a winner.
Choosing a higher confidence level is often preferable when testing a variable that only slightly improves the conversion rate. The random variant is then likely to play a larger role.
For example, it would be safer to lower the confidence threshold in the case of an experience that will likely increase the conversion rate by 10% or more, such as a revamped testimonials section.
In short, the more specific the change to be expected (color of a button, small piece of text, etc.), the less the impact on the conversion rate will be. In this case, a scientific approach should then be adopted.
Define the duration of the test
Your test should take place over a sufficiently long period of time for you to obtain a large enough sample.
Determining the duration of a test can be more complex than it seems. Depending on your business and how the A / B test is run, it may take a few hours, days, or weeks to reach a valid conclusion.
One of the biggest factors is the volume of traffic you generate. If the traffic to your business website is not high, the A / B test will take longer to run.
Rely on user feedback
A / B testing is mostly about quantitative data, but it doesn’t necessarily help you understand why people take certain actions over others. When doing an A / B testing, consider collecting feedback from target audiences.
The survey or poll is one of the best ways to question user habits. Add a survey to your site to ask visitors why they didn’t click on a certain CTA, or on your thank you pages to ask them why they clicked a button or filled out a form.
You may then find that a large number of users have clicked on a call-to-action redirecting them to an e-book, but they didn’t convert due to its price. This kind of information will allow you to understand the typical journey of your users.
To go further, download this free guide and find out how write your emails to maximize their impact and boost your brand growth.