One of the main issues that people have with A/B testing is that it can often take a long time to see results. It’s fine if you’re Amazon and your site gets millions of visitors a month but if you run a smaller site, or you want to test one of the pages within your site that gets less traffic, then it can take a long time to find a winning variation. This is even more of a problem if the changes you’ve made are quite minor. While small changes can make a big difference, more often than not small changes generally mean a small difference to your results. Running tests with minor changes can quite often be inconclusive, no matter how long they are run for.
Another area to look out for when you’re running your tests is cloaking. Cloaking is a black hat SEO technique where the content shown to a search engine spider is different from the version that’s shown to a user. By showing different versions of a page to different people A/B testing runs the risk of being seen as cloaking by search engines. This shouldn’t be a problem as long as you follow the official Google Guidelines, but it is something you should consider if you’re planning on running tests.
A problem with relying on quantitative testing is that this often leads to a ‘conversion first’ mind-set and can dehumanise your users, turning them into little more than lab rats in your experiments! A conversion driven mind-set makes business sense but it should not be at the expense of the overall user experience. Focusing purely on conversions can lead to short term ‘fixes’ that end up doing more harm than good in the long term.
There’s also some difficult ethical questions around purely focusing on making your users convert. The difference between using Dark Patterns (user interfaces which are designed to trick people into taking actions that they did not mean to take) in your design and doing conversion rate optimisation.
“Conversion optimisation should be a win-win situation with users having a better purchasing experience and website owners making more sales. Tricks and tactics that are used to prize more money from users against their will may lead to an increase in online sales, but will almost always be outweighed by the costs of additional customer service to deal with complaints as well as the negative reputation that dark patterns will bring for your brand.”
There really isn’t anything to be gained in the long term by using Dark Patterns in your design but there is a temptation to move towards some of these underhand techniques when A/B testing. In the short term, and in isolation, they are likely to lead to higher conversions, but normally at a cost.