The Cost of Sampling in Google Analytics
Weber shows that while sampling works well for overall trends on your website, it can become very inaccurate. One good example of this is when you’re looking at smaller subsets of data, such as the conversion rate of a specific campaign.
To help calculate the accuracy of your data, Weber even included an online tool, which you can use to determine the “margin of error” for your campaign conversion rate.
In this post, I want to take his argument a bit further. I wanted to ask a slightly different question and see what the sampling inaccuracy could actually COST you, particularly if it led to taking the wrong decision in a campaign.
First, calculate margin of error
Imagine that you have a website with some 10M sessions in a particular time period, and that sampling kicks in. Now, suppose you’re comparing five different online campaigns, each of which is reported to send 4K sessions to your website with conversions of 2%.
Using Jonathan’s tool, linked above, we can now calculate the margin of error for these reported conversion rates. Here is the result:
As you can see, even though Google Analytics reports a conversion rate of 2%, the real rate for each of the five campaigns is actually between 0.55% and 5.04%. A significant difference, whichever way you look at it.
Next, calculate sampling cost
Let’s assume you have invested $6K in each of the five campaigns (for a total cost of $30K). Let’s then say that each converted session is worth around $100, making your total revenue from the campaigns $40K. This gives you a total profit of $10K:
Now assume that Campaign A is doing much better that the rest. Assume that its conversion rate is 5.04%, while the other campaigns only convert to 1.24%:
In this case, you would still see a profit of $10K across all your online campaigns, but you would only profit from Campaign A.
Now imagine that you have an additional $30K to invest in the five campaigns. If you allocated this money based on GA’s sampled report, you would still end up with $10K profit.
But, if you had had access to the unsampled data, you would have known that Campaign A is the only profitable one, and that investing the entirety of your $30K into this campaign was the smart move.
Indeed, this simple decision would net you nothing less than $70.8K in profit:
As you can see, sampling in this hypothetical case would have cost you $70.8K – $10K = $60.8K.
That’s not small change by any means. Think about that next time you run into this seemingly innocuous message:
Unsample your data with Scitylana
The potential cost of sampling in Google Analytics for business is one reason we created Scitylana. This tool lets you extract data from your free Google Analytics account, transforming them into a raw, hit-level form.
The data is free from sampling, giving you more freedom to analyze, visualize, and integrate GA and CRM data.
The best thing? Signing up is absolutely free, and you can do it right now in just a couple of seconds: www.scitylana.com