How to start A/B testing and not regret it
For me A/B testing shows that someone is taking data seriously. An organization needs maturity with data handling operation to be able to reap the benefits of A/B testing. Otherwise it can only have a negative effect on daily operations as it will raise the running costs of the analytics department and the confusion between the business stakeholders.
On the other hand, if you have reached a state where the analytics business stakeholders have clearly defined KPIs, spend time discussing website optimization goals and the data collection setup is working consistently, then definitely A/B testing is the next step.
This slide deck was originally presented in the Digital Analytics meetup sessions in Thessaloniki, Greece. This is a digital analytics community, which I started in 2017 and now consists of about 1600 members.
This deck aims at:
- introducing digital analysts to the basic concepts of A/B testing
- helping digital analysts understand how it can empower a web site’s/mobile app’s analytics team
- explaining the complexity of A/B tests and the level of detail you have to go into, in order to have meaningful results
- explaining the use cases, where an A/B test process can be useful, even beyond web analytics
Knowing all the points above is imperative, if you don’t want to regret getting into this time-consuming process. If you checked all the boxes and you feel that you have what it takes, the next steps is to choose the right tool for the job. You will find in this deck a list of the most common features found in A/B testing tools (i.e. Reporting, Behavioral targeting etc.) and detailed comparison of the most common tools in the market (both of smaller and larger scale).
Finally, since one of the things I enjoy the most in A/B testing is proving myself wrong using data, you will find a few examples of successful A/B tests in different organizations, showing the different variations and the one which increased the website’s performance towards the organization’s KPIs.
How to start ab testing and not regret it from Panagiotis Tzamtzis
If you know Greek, you can also check a recording of the complete presentation, from the Digital analytics meetup Thessaloniki session (up to the 32nd minute):
