Why Impression-based Metrics Are Misleading

Profit per Impression, or PPI, is a popular metric in PPC. So popular that it sparked a little side discussion during last week’s #ppcchat when I called it misleading. Let me explain why.

As a metric, PPI is used to evaluate the performance of ads. It includes metrics such as click-through rate, conversion rate and average order value, making it look like the ultimate metric to evaluate an ad. I don’t think it is.

Per-impression metrics

The argument I’m going to make actually applies to all “per-impression” metrics in PPC. Whether its profit per impression, conversions per impression or clicks per impression (otherwise known as CTR): They all share the same flaws.

Basically, all of these metrics are something good per impression . As a fraction:

per-impression metric = something good / impressions

All of these metrics are considered better when higher: A higher click-through rate is better than a lower one. A higher profit per impression is better than a lower one. Sounds right, doesn’t it?

One of the problems is that profit per impression also goes up when the number of impressions goes down. More importantly, the metric also goes up when profits decrease, as long as the impression count decreases even more.

The idea here is to use the number of impressions as a baseline. This implicitly assumes two things:

  1. The number of impressions is an independent variable for the ad.
  2. All impressions are equal.

Both of these assumptions are wrong.

Assumption 1: Independent impressions

To illustrate where PPI can be misleading, let’s look at an example. Let’s say we did an AdWords ad test in an ad group with two ads, set to rotate evenly. This could be the result:

Example data for an A/B ad test

Which ad would you prefer?

Looking at profit per impression, the answer is clear: A had a profit of $1 per impression whereas B only made 75 cents per impression. According to PPI, A is clearly better. However, B made a higher profit.

Here a common misconception comes into play. These ads were supposed to rotate evenly, but B got more impressions. This seems like a contradiction – if B got a much higher number of impressions, then the rotation couldn’t have been even. With this in mind, PPI makes sense since it accounts for an uneven rotation.

The problem is that ad rotation actually refers to auctions, not to impressions. Even rotation means that both ads are supposed to enter a similar number of auctions. An impression only occurs if an ad is ranked high enough to be displayed on the results page.

In the ad auction, an ad’s quality is an important factor. With everything else the same (bids, ad extensions), the number of auctions that leads to an impression heavily depends on the ad itself. This contradicts assumption 1 (the number of impressions being independent from the ad).

What does that mean for our example? It means that the higher impression count of ad B is actually because B was more often successful in ad auctions. It means that the ad rotation might actually have been even, despite the different impression counts.

Given an even ad rotation, A and B got the same chance (50% of auctions each). Ad A made $5,000, B made $6,000. Looking at profit instead of profit per impression leads us to declare ad B the winner.

But…

You may have noticed: Ad A had a higher click-through rate than B: 10% compared to 7.5%. Conventional wisdom tells us that CTR is highly correlated with quality, which would contradict my claim that B was the higher quality ad. Again, there is a misconception. Conflicting with the second implicit assumption for per-impression metrics is the fact that impressions are not equal at all.

Assumption 2: Impression equality

An ad impression has many different aspects such as the query and the device it’s coming from. Let’s stay with the query for a moment. Expanding our example from above let’s say there are only two queries: a very relevant one and a less relevant one:

Some example queries for our test

As you can see, ad B does a little better in both cases: Its CTR is always a little higher.

In the ad auction, these different CTR’s translate to different quality scores, which in turn lead to different ad ranks. In this example, both ads have no problem to be displayed for the highly relevant query, but A can’t make the cut to be displayed for the less relevant query. This leads to B having impressions for the less relevant query, too, lowering its average CTR. A on the other hand is unburdened by the low CTR query.

From the outside – from the perspective of an account manager – the distinction between more and less relevant queries cannot be made. All that’s visible is that ad B has a lower CTR and a lower profit per impression.

Reality check

Sure, this was a very simple example. In reality, there are much more aspects to a search that triggers an auction or impression. Some of them we deal with every day, like queries or devices. But much depends on the searcher:

  • Is this the first time this user searched for this query?
  • How did he or she react to our ad and similar ads in the past?
  • Does this person avoid to click on ads in general?

All of these are important factors that should be considered when evaluating an ad. Google uses information like this (and more) for both the ad auction and optimized ad rotation. The latter is part of the reason why Google is able to make decisions about optimizing ads much faster than we are.

As outsiders with hardly any data we can’t always understand how Google reached a decision. We tend to think that the metrics we have are all there is, but there are many more things at play than we realize.

Conclusion

Whether it’s the all-encompassing profit per impression or a basic thing like CTR: Impression based metrics aren’t as reliable as we’d like them to be. This doesn’t mean they’re completely useless, especially since we don’t have more reliable alternatives. However, they shouldn’t be treated as absolute truths because they aren’t.