Behind the scenes of A/B testing: benchmarks and insights from three digital publishers

Credit: Photo by Letizia Bordoni on Unsplash

Optimising your paywall and subscription strategy to compete for a place in your reader's monthly budget is no mean feat. And whilst there are endless benchmarks and success cases online, there is really only one way of finding the best performing strategy for your unique audience: testing.

Being one of the most user-friendly but efficient ways of optimising your paywall performance, testing is something that is highly recommended for publishers to do on a regular basis and in a variety of forms. 

But, unlike many of the other optimisation methods, it is not the easiest to benchmark given that the whole point of running an A/B test is that users are unaware when one is taking place. 

Here are three examples of paywall A/B tests run by digital publishers - what can you test, what do you need to define prior to launching the test, what were the results and, most importantly, what can we learn from these to apply to your own tests? 

Bio à la Une 

This French health and well-being publisher, who only recently launched a premium strategy, wanted to learn more about their audience and what makes them convert. They, therefore, ran a series of tests, one of these varying the wording on the call-to-action button. 

Hypothesis: Click-through rates (CTRs) will increase by using a more direct call-to-action phrase, "je m'abonne!" (Subscribe), one tha is commonly employed by digital publishers in France.

Variable: Text on the CTA (call to action) button.

Goal: Increase CTRs and discover the focus that works best for each audience segment.

Control (version A): CTA button text "Discover our offers", a more invitational, indirect phrase.

Treatment (version B): CTA button text "Subscribe" takes a very direct approach.

Results: Version B had a higher click-through rate amongst less engaged users, which suggests that these audiences respond better to a direct CTA phrase. Version A, which invites readers to discover subscription offers, performed better for more engaged users. 

This was an interesting result, and one that highlights the importance of segmenting your audience before testing. In this instance, the publisher used Poool’s Dashboard to segment users based on level of engagement (Volatiles, Occasionals, Regulars and Fans) which gave insights into how these users respond differently to direct vs indirect wording on the paywall. 

To further optimise your paywall performance and gain a deeper understanding of user behaviour, you could also consider segmenting audiences based on location, device or even a users interests if you’re able to gather this information. 

The takeaway: Just as the performance of a paywall will differ on your site compared to that of another publisher, this can also be said for one segment of your audience compared to another. 

ELLE Magazine

The fashion and beauty magazine launched its digital premium strategy with a two-step conversion journey consisting of a registration wall followed by a hard paywall. This proved hugely valuable for increasing engagement and informing their audience of the concept of paying for access to content, but the team wanted to test a more direct approach to see if this led to higher conversion rates. 

Hypothesis: A hard paywall will lead to higher conversion rates compared to a soft engagement journey.

Variable: User journey.

Goal: Increase conversion rates.

Control (version A): A two-step scenario with a soft conversion registration wall followed by a paywall.

Treatment (version B): A hard paywall alone.

Results: Version B led to slightly higher conversion rates than version A. However, these results have to be taken with a pinch of salt…

Firstly, it is important to consider the stage of your subscription model. For ELLE, although version A worked extremely well for launching its strategy, collecting data and increasing engagement prior to subscription, version B performed better when this test took place, meaning their audience was perhaps more qualified and less likely to be frustrated by a hard paywall. 

You also have to consider more than simply conversion rates as a KPI (key performance indicator) when altering your user's journey, namely ARPU (average revenue per user). Considering engagement has a direct correlation with revenue, and that soft conversion steps such as registration walls increase user engagement, it is arguably more beneficial in the long term to employ a user journey such as version A, despite what these test results suggest. 

Specifically, de-anonymising a user through registration increases ad-based revenue thanks to targeting and also boosts propensity to subscribe (and increases the likelihood that this user will remain subscribed in the long run). 

The takeaway: When looking at benchmarks, do not forget to consider the context surrounding them, take overall ARPU into account and remember that it is not only the paywall that influences conversion rates.

For instance, although ELLE did choose to employ journey B on all user segments, it also worked on increasing the visibility of premium content, reducing frustration by adding yellow tags on these subscriber-only articles as well as boosting engagement in other areas, such as by promoting their newsletter and membership offers. These efforts had a big impact on balancing frustration and engagement which ultimately support high conversion rates.

Neos Kosmos 

One of the most important aspects of your premium model is the value proposition. Why should a user subscribe? What pain points will it solve? How will it help the reader achieve their 'jobs-to-be-done'? 

And, of course, this value proposition needs to be clearly communicated to readers on your paywall in a way that is easily understood and gives enough encouragement to click-through and subscribe. 

This is something that the Australian-Greek publisher Neos Kosmos understands well, leading them to test out a shorter version of their value proposition which focuses more on 'supporting' the publisher than on the benefits of subscribing.  

Hypothesis: A value proposition focusing on 'supporting' Neos Kosmos will lead to higher conversion rates. 

Variable: Wording on the wall (value proposition).

Goal: Increase conversion rates.

Control (version A): The original wording details what Neos Kosmos' premium offer gives a subscriber access to.

Treatment (version B): This version takes a more 'support our work' approach, appealing to a user's emotional and cultural attachment to Greek culture in Australia.

Result: Version B led to a 16 per cent increase in conversion rates. 

For Neos Kosmos, this showed the importance of appealing to their audiences' cultural heritage and the shared values amongst readers. For many publishers, taking the angle of asking for support rather than providing access to content is hugely valuable for increasing conversion rates and developing a community of loyal subscribers. 

The takeaway: Value propositions need to be clearly defined prior to launching your premium strategy but should also be tested on a frequent basis to discover the optimal angle, wording, etc. for your unique audience.

To summarise:

• Clearly define your hypothesis, variable, goal and audience segments prior to launching the test
• Remember that there is more at play than just the paywall, such as value proposition, visibility of premium content, the balance between frustration and engagement, etc. 
• Tracking click-through and conversion rates alone has its limitations - ARPU is arguably the most important
• As always, benchmarks are great and useful for inspiration, but there is no one-size-fits-all strategy that will work for every digital publisher

Happy testing.

Madeleine White is the content marketing manager at Poool, the audience conversion platform for digital publishers to convert anonymous users into leads, members and subscribers. Her work focuses on revenue, engagement and conversion strategies for digital content producers.

Free daily newsletter

If you like our news and feature articles, you can sign up to receive our free daily (Mon-Fri) email newsletter (mobile friendly).