Apples to Apples and GA4

Often, your data is not inaccurate. It's accurately telling you something other than what you think it's telling you and leading you or your audience to draw the wrong conclusion.

As we read in Chapter 6 – Plan, Plan, Plan, we need to have our own dimension and metric libraries to ensure that we don't get lost in the actual definitions, and as we saw in Chapter 1 – Climbing the AMC, we need to be sure that data integrity checks and active reconciliation (detailed comparing of like-for-like metrics) are part of our audits. The only sound way to complete a proper data integrity check (apart from regression testing from a previous benchmark) is to compare data sources against each other, and that can often mean comparing MarTech tools and reports like e-commerce reporting to back-end financial systems and accountancy programs that throw up unusual results. It's at this point that we can lose confidence and start worrying that our data might be F**ked.

If we compare the Sessions we see in our web analytics tool with the Visits reported on our website (often built into the CMS back-end), we will see that these are completely different. It doesn't necessarily mean that one of the sources is wrong. They could measure two different, similar-sounding metrics correctly but count them in different ways.

Often 100% accuracy is not possible – in Chapter 4 – Cookie Apocalypse, we explore why, due to the limitations of our tools, we should only expect some metrics to be 95% accurate anyway. The Revenue metric we measure in Google Analytics is an example; a 5% variance or less to the revenue recorded in your bank account is more than good enough.

Several reasons can explain an unexpected significant variance in two metrics without their being a problem. Here are some key overlapping factors that can contribute to the mismatch:

As we'll see, Google Analytics 3 collects and then processes data differently from Google Analytics 4.

This was a key reason why websites used to regularly over-report Visitors. Google Analytics had much better access to information about the most recent bots (false traffic created by spammers), which they used to filter out fake sessions automatically.

A common mistake is comparing Visits to Sessions or Users to Sessions. A user has a session on a website and opens a page, which is a Page View. These are all different scopes or hierarchies of data.

This is again different between Google Analytics 3 and Google Analytics 4.

You can control This part of the process by creating and maintaining your own data dictionaries and libraries and sharing them for everyone to review.

We're likely to see this misunderstanding get worse rather than better with the introduction of Google Analytics 4. A GA3 Session is calculated differently from a GA4 Session. They are not the same metric but have the same name, and there are many more examples of these differences. Here are those most likely to cause problems for us marketers.

Key Metrics:

Key Dimensions:

Our example here uses Google Analytics as it's such a common tool and just about to go through (depending on when you read this) a huge upheaval.

It is essential to carefully evaluate and understand these factors for each of the different tools that we use, as this is often the only way to identify the root causes of data mismatches and ensure reliable analysis and decision-making. We must ensure we don't mislead naïve users with poor naming, definitions, and labelling, and we can ensure this by both maintaining and auditing our data and libraries.

As we explored in Who Wants Data, we may also need to restrict access to raw data and platforms, ensuring that we only provide guided 'Knowledge and Wisdom' along with the metrics and dimensions we show in our prepared dashboards and reports. This is why we don't leave the analysis to the executives (a key principle suggested in Plan, Plan, Plan). They don't have the time to understand why and how two metrics with the same standard name, from the same tool, with everything working no longer match and why in this previously specialist field that's OK and not completely insane or illogical.