Statistically significant, that is.
Whenever you are running any kind of test, you must let the test run long enough so that you can trust the results you are getting.
Why is this?
Because if you run two versions of the exact same ad in online advertising, to the exact same people, one of the ads will initially outperform the other by a large margin.
You have to have a large enough sample size to be able to determine whether your results are meaningful or not.
Things that can mess up your results:
- Time of day the test was run
- What day of the week the test was run
- The time of year
The first thing you can do to address is this, is to test both of your options at the same time. This is a starting point to make sure your results are on even ground. However, it is possible that your first advertisement will perform better on the weekend and your second advertisement performs better on weekdays. That is one reason you may want to run your two advertisements for at least a week before you see which one performs better.
Another thing you will want to consider is that if you are introducing a new advertisement where you used to run the same old one, there can be a novelty factor for the first while it runs. It may outperform the original for a short time because of the novelty factor, but in the longer run the old advertisement could be the better performer.
Where this whole process gets really tricky is when you don’t have enough numbers to actually know if your results are statistically significant.
I think this is where you have to make a qualitative decision instead of a quantitative decision.
It turns out, that in business, a lot of your business decisions will need to be made on a qualitative basis.
Qualitative decisions can be made on:
- Feedback from very few people (clients I mean, not your friends)
- If you are running two ad campaigns and they are both performing the same
- Which one contributes more to your strategic objectives?
- Which one projects the brand image that your company wants to portray?
- Was one process more efficient than the other?
- How you feel. Yes, I said it: how you feel. Keep in mind, this is in the absence of available quantitative data. But often, you need to make decisions on what you feel improves the process, marketing, training, or whatever it is.
Bottom line: if you are making quantitative decisions, then make sure your results are statistically significant. If you are making qualitative decisions, then know what your criteria for making the decisions are. Don’t base your qualitative decisions based on half-baked quantitative data.