Keyword Quality Score Up Close

Google has just released a new version of the AdWords API. There’s nothing spectacular – unless you’re a Quality Score geek.

The API version v201601 comes with three new fields for the KEYWORDS_PERFORMANCE_REPORT :

  • SearchPredictedCtr
  • CreativeQualityScore
  • PostClickQualityScore

You’ve seen these three before:

qs factors

What gets me excited is the chance to get these three factors into a report, allowing for further analysis on a massive scale. So I downloaded a report with a lot of keywords and analyzed the results in Excel.

Results

qs matrix

I looked at the combinations of the three factors and what Quality Scores they lead to. The complete results are shown on the right. A plus stands for above average, a zero is average and a minus means below average.

The first thing I found was that each combination of the three factors always leads to the same Quality Score. For example, if expected CTR and ad relevance are above average and landing page experience is just average (+ + 0 in the table), that’s always an 8. There were no exceptions.

What’s more interesting is that ad relevance is less important than the other two factors. For example, a 10 requires all three factors above average. Expected CTR or landing page experience going down to average leads to an 8, whereas losing ad relevance still gets you a 9.

More important is the difference when you look at it from an optimization perspective: Improving ad relevance by one (from below to average or from average to above) always improves Quality Score by one. Improving expected CTR or landing page experience on the other hand will often improve Quality Score by two points. Both factors seem to be equally important.

In any case: Improving one of the factors will always improve Quality Score.

Weights

Treating the table with its 27 rows as 27 equations it wasn’t too hard to find weights for each of the three factors.

Each step in ad relevance (from below average to average or from average to above average) changes Quality Score by exactly one point. For expected CTR and landing page experience every step changes Quality Score by 1.75 points.

Using these weights of course leads to some decimal numbers. For example, with all factors average, you get a Quality Score of 5.5 (right in the middle between 1 and 10). But rounding these numbers always leads to the correct Quality Scores (see table below).

Interpretations

There are a few ways to interpret these numbers. Apparently, expected CTR and landing page experience are 75% more important than ad relevance. We can also give in to temptation, translate these weights into percentages and make a nice pie chart:

pie

Consequences

qs matrix decimalsUsing different weights for the factors has some interesting implications: It gets us more scores than just the integers from 1 to 10 (as shown in the table on the right). Instead, we can get 15 different scores. At the same time, not all integers are valid: There’s a 3.75 and a 4.5, but you can’t use the weights to get to a 4.0.

Using the numbers from the report we could calculate more accurate Quality Scores. This could be helpful for tools that report on Quality Score, but there’s little benefit in the added decimals for the day-to-day account management.

For optimization purposes this can also help since it tells you to work on CTR and landing pages rather than ad relevance. However, in practice, CTR and ad relevance are closely related and there’s no reason to improve one without the other.

The Disclaimer: Auction Quality Score

While I enjoy diving into keyword Quality Score, I can’t emphasize this point enough: Keyword Quality Score is not the real thing. Keyword Quality Score is a number between 1 and 10 and it’s obviously made up of three factors with three possible values each. What Googles uses in the auction is much more complex.

I’ve long been saying that auction Quality Score is Google’s best estimate of click-through probability (a.k.a. exptected CTR). I don’t think landing pages are considered in the auction at all, but if they are, they’re pretty unimportant (certainly nowhere near 39%).

I don’t want to go into the details again, but be advised that keyword Quality Score has a few shortcomings by design. A big one is its limitation to a maximum of 10, which gives people the impression that there’s nothing to be gained by further optimization. With the three factors the limit is above average, which sets the bar even lower.

I believe this can lead to the wrong conclusion: Just because some visible values can’t be improved any further doesn’t mean you should stop optimizing.

If expected CTR is above average – great. But that doesn’t mean that you’re done.

Update 1: Look at your own data

Google has added support for this kind of report in AdWords Scripts. With the following script you can create such a report and export it to a spreadsheet. Just run it and look for the spreadsheet’s URL in the logs.

function main(){  
  var spreadsheet = SpreadsheetApp.create("Report");
  var report = AdWordsApp.report("SELECT CampaignName, AdGroupName, Criteria, KeywordMatchType, QualityScore, SearchPredictedCtr, CreativeQualityScore, PostClickQualityScore FROM KEYWORDS_PERFORMANCE_REPORT WHERE Id NOT_IN [3000000, 3000006] AND Status = 'ENABLED' AND AdGroupStatus = 'ENABLED' AND CampaignStatus = 'ENABLED'");
  report.exportToSheet(spreadsheet.getActiveSheet());
  Logger.log("Report available at " + spreadsheet.getUrl());
}

Update 2: SEL Post

A very similar analysis has been posted by Brad Geddes on SearchEngineLand. I’m still waiting to hear back as to how this happenend.

About Martin Roettgerding
Martin Roettgerding is the head of SEM at SEO/SEM agency Bloofusion Germany. On Twitter he goes by the name @bloomarty, and you can find him regularly on #ppcchat.

  • mtg

    Could you describe better auction Quality Score?

  • Andrew Goodman

    Great work. I’m glad you continue to emphasize the following: “While I enjoy diving into keyword Quality Score, I can’t emphasize this
    point enough: Keyword Quality Score is not the real thing. Keyword
    Quality Score is a number between 1 and 10 and it’s obviously made up of
    three factors with three possible values each. What Googles uses in the
    auction is much more complex.” IMO that basically means, the “real” impact to Quality Score is weighted much higher on CTR than this exercise reveals. (“Reveals” being the wrong word, actually, given that it is crunching front-end numbers that Google admits aren’t the real numbers, just very broad summaries of what’s under the hood.)

    • Martin Röttgerding

      Thanks, Andrew!
      I wonder if we can reveal a bit more about the relationship between CTR, the visible CTR expectation, and QS. Expect another blog post some time 🙂

  • I am glad to see ad relevancy ranking last in qs importance. This does make complete sence however as I can often write a killer ad that will get a very high CTR, but won’t be relevant to the keyword and should not be a huge limitation all on its own. “Keyword Relevance” is more like a quick way for Google to evaluate QS before having any data to work with.

  • Lukas Mehnert

    Hi Martin,

    this is a very old picture about QS from a Google Engage conference in Spain.

    But maybe it is still true.

    Thanks for your interesting find.

  • Diogo Abrantes da Silva

    Hey Martin I have a keyword with a QS 8 with a “below average” CTR (-,+,+). Is it possible that you’re missing some row’s on your table? Or it depends on the account?

  • Martin Röttgerding

    Hi Diogo,
    Regarding my data, the table is complete. As I wrote, “The first thing I found was that each combination of the three factors always leads to the same Quality Score.”

    I rechecked my data with a smaller sample (50,000 keywords) to see if anything changed. A picture of the pivot table is attached – so far the results are the same as before. That said I can’t guarantee they’re universally correct. Is your 8 with -++ something you see more often?

  • Atif Salim

    Hi Martin,

    Thanks for the brilliant insight & analysis.

    Just to let you know that I updated your adwords script to only show the quality score factors for keywords which received an impression. For some large ppc accounts I was getting an error that the spreadsheet would be over the 200000 row limit. The impression filter helps to get around that.

    function main(){

    var spreadsheet = SpreadsheetApp.create(“Report”);

    var report = AdWordsApp.report(“SELECT CampaignName, AdGroupName, Criteria, KeywordMatchType, QualityScore, SearchPredictedCtr, CreativeQualityScore, PostClickQualityScore, Impressions FROM KEYWORDS_PERFORMANCE_REPORT WHERE Id NOT_IN [3000000, 3000006] AND Status = ‘ENABLED’ AND AdGroupStatus = ‘ENABLED’ AND CampaignStatus = ‘ENABLED’ AND Impressions > 0”);

    report.exportToSheet(spreadsheet.getActiveSheet());

    Logger.log(“Report available at ” + spreadsheet.getUrl());

    }

    • Emily Griffiths

      Hi Atif,

      Really useful! What date range does this script look at?

      Thanks,

      Emily

  • Jered

    How would I edit the script to behave like your New QS script that creates a new set of folders to organize and share the files with clients. The QS script 3.0 is great and I want to try to utilize some of the great features within this script.

  • Ali

    Hi Martin,

    thanks a lot for sharing.

    Is it possible to add Impressions/Clicks/Cost to the table. Makes it easier for me to prioritize where to improve next.

    Best

    • Martin Röttgerding

      Hi Ali,
      Yes, absolutely. You can add fields to the report by adding them in the SELECT part… use this line instead of the the third one above:

      var report = AdWordsApp.report(“SELECT CampaignName, AdGroupName, Criteria, KeywordMatchType, QualityScore, SearchPredictedCtr, CreativeQualityScore, PostClickQualityScore, Impressions, Clicks, Cost FROM KEYWORDS_PERFORMANCE_REPORT WHERE Id NOT_IN [3000000, 3000006] AND Status = ‘ENABLED’ AND AdGroupStatus = ‘ENABLED’ AND CampaignStatus = ‘ENABLED'”);

      • Emily Griffiths

        Hi Martin,

        This is great. How can I add for it to pull in the KW level URL in with the keywords performance report?

        • Martin Röttgerding

          Hi Emily,
          Oops, I forgot about that (wasn’t needed for the original QS part) – you need to add this to the query as well:

          var report = AdWordsApp.report(“SELECT CampaignName, AdGroupName, Criteria, KeywordMatchType, QualityScore, SearchPredictedCtr, CreativeQualityScore, PostClickQualityScore, Impressions, Clicks, Cost FROM KEYWORDS_PERFORMANCE_REPORT WHERE Id NOT_IN [3000000, 3000006] AND Status = ‘ENABLED’ AND AdGroupStatus = ‘ENABLED’ AND CampaignStatus = ‘ENABLED’ DURING 20160501, 20160531”);

          The date range is in the last part, the DURING clause. In this case it’s an absolute date range. You can also use the relative date ranges that are available in AdWords. They may be a bit cryptic… here’s what’s available: TODAY, YESTERDAY, LAST_7_DAYS, THIS_WEEK_SUN_TODAY, THIS_WEEK_MON_TODAY, LAST_WEEK, LAST_14_DAYS, LAST_30_DAYS, LAST_BUSINESS_WEEK, LAST_WEEK_SUN_SAT, THIS_MONTH

          example: “(…) DURING LAST_30_DAYS”

          HTH

          Martin

  • sem champ

    Outsource PPC
    it is very important to increase the quality score and this article was really helpfull.