A/B testing
The importance of A/B testing and how I use it in my work
What did I learn about A/B testing?
A/B testing has allowed me to make careful changes to their user experiences while collecting data on the results. This helps me to construct hypotheses and learn why certain elements of my client's experience impact user behavior. In another way, when I give them recommendations of new designs, correction of current designs, this is a way to back up my recommendations, or sometimes, learn about why certain things don't work for certain clients.
The reason why a lot of common practices work in some clients, and in others don't is because of the type of audience, the market, the type of business, the current traffic, and many other elements.
Every business is unique, it will be impossible to predict how the changes will impact the site without testing them.
​
How to define a strong Hypothesis?​
​
A strong hypothesis consists of:
1. A description of the change to be made
2. The expected effect the change has on specific metrics
3. A rationale that explains why the effect follows the change
How do I get clients on board to test all the recommendations and not implement them straight away?
Before presenting my new ideas, I always give them a quick introduction or refresher to the best practices of A/B, and explain the importance of performing those tests before implementing them.
Is A/B testing suitable for my client?
If the traffic and transactions are low (lower than 1000 transactions per month) on the site, we should go with other methods than A/B testing.
Instead, I would consider qualitative research methods to gather optimization insights such as:
-
Customer survey
-
User testing
-
Usability evaluations
Things to know before A/B testing:
On average 30% of A/B tests are successful. This means that 70% of results will have no impact at all, or will have a negative impact on the site.
Negative test results aren’t failures, and when I get a negative result it only confirms the importance of that test and helps me to choose the next steps and what to avoid going forward.
Priorities
​
Avoid testing everything at one time.
It makes it hard to assess impact.
There are 3 main things that you should consider when prioritizing:
-
Potential: How much improvement you expect?
-
Importance: How valuable is the traffic of that page?
-
Effort: How difficult is to implement?
To support my clients with this, I usually provide an excel google sheet "impact tracker" to track all the tests that I have suggested. In this sheet, I will add two columns to assess:
-
Potential business impact
-
Expected ease of implementation
.png)
These two columns are only a guide for the client, of course, I can't promise the impact or the ease of implementation to be accurate as this will depend on the structure of the team, and the developers and analyst experience.
Apart from guiding the client with the test prioritization, this trix is live, making possible to see changes and communicate without having to send numerous emails, as there is columns of communication like "Status" and "Comments", as well as "Owner of the test".
How Long should I wait to see the test results?
The most common mistake that is being done in A/B testing is either stopping a test too soon or too late.
In order to avoid this, you should be:
-
Calculating the sample size for the test
-
Waiting for two business cycles
(two weeks at least) -
Assuring that the results are statistically significant
​​
Rule of thumb: Don't declare winner until: 200 conversions, 14 days & 95% confidence.
Example of client positive A/B test:​
Before:
​
I recommended making the search bar bigger and brighter.
A way to make it more predominant will be inverting the colors from white background and black symbols and text.
After
The new search bar was tested, with two A/B test in place resulting in positive results:
​
Significance increase:
-
“CTR Search” 99.94%
-
“Search Open game logged in” 98.95%
Increase​ data:
-
“CTR Search” 10.22%
-
“Search Open game logged in” 8.40%
Increase of conversion rate:​
-
“CTR Search” 27.40%
-
“Search Open game logged in” 23.49%
