A/B testing – making changes to a marketing communication (website, video, advertisement, etc.) and seeing whether option A or option B garners better feedback from an audience is fraught with pitfalls. But it’s also sometimes your best option for gaining clarity – and important information – for decision-making purposes.
It is important to understand that A/B testing is not a silver bullet – in fact, it’s possible to spend the time and money on this and the results don’t give you anything meaningful. We use it when we want to see which marketing communication performs better – not best. There is no “best” because you cannot test every possible permutation.
If we go forth with A/B testing – perhaps because we want more than our own opinions – we must remain objective. The results are a data point to inform our decisions, but could also be misleading, blind to critical factors, lack statistical significance or rely too heavily on it. There are actually over 20 observable problems with A/B testing. This means we should not expect to derive “best practices” from our testing – nor truisms, axioms, clichés, etc.
Sometimes, we will have more questions than we began with – we should not automatically expect A/B testing to generate answers.
So, proceed with a balanced approach and you may glean vital insights AND avoid the many traps.
Like…red pill or blue pill? Have fun!
