Is NPR biased

New study: Is Google biased on certain media sites

Recently, Google's CEO Sundar Pichai had to testify in front of Congress about possible biases regarding Google's algorithm. It's not the first time Google has been accused of bias, and it probably won't be the last. While Google affirms that there is no bias, many conservatives claim that Google is biased against them.

Given our expert knowledge of Search Engine Optimization (SEO) and Ahref's large amount of data, we wanted to see if we could identify any bias by examining data from popular conservative and liberal media sites. In case you are not familiar with Ahrefs, we are one of the leading SEO tools with significant amounts of data on the web.

Google makes hundreds of changes to its ranking algorithm every year. Most of them go unnoticed because they are marginal, but every now and then a major core update is made that affects a large part of the search results. Since Google is telling us the date these updates will be made, we figured we could determine bias by examining organic traffic to popular liberal and conservative media sites before and after these updates.

For example, here is the estimated organic search traffic on Fox News since 2015. Each dashed line represents a Google Core update:

However, we can hardly draw any conclusions if we only look at this data for a website, which is why we also used the data for the best-known conservative and liberal media sites. We have compiled these with the help of AllSides Media Bias Ratings (classification of the situation on the political spectrum). Here is a list of the websites:

Conservative media sites:

  • New York Post
  • The Last Refuge
  • Drudge Report
  • The Federalist
  • Orange County Register
  • The Epoch Times
  • Washington Times
  • Christian Broadcasting Network
  • National Review
  • Townhall
  • The Mark Levin Show
  • The Rush Limbaugh Show
  • Breitbart
  • Newsmax
  • The National Interest
  • The Gateway Pundit
  • RedState
  • PJ Media
  • Washington Examiner
  • Fox News
  • Christian Today
  • Zero hedge
  • The Daily Caller
  • TheBlaze
  • The Daily Wire

Liberal media sites:

  • Vox
  • U.S. News & World Report
  • The Washington Post
  • CNN
  • Bustle
  • NBC News
  • Hollywood Reporter
  • Los Angeles Times
  • Yahoo News
  • Al Jazeera
  • Rolling Stone
  • HuffPost
  • The Verge
  • The New York Times
  • ABC News
  • TIME
  • CBS Local
  • The Guardian
  • Bloomberg
  • NPR
  • CBS News
  • The Atlantic
  • Politico
  • Univision

Before we get into the results, I should give a little more context on Ahref's data. We have hundreds of millions of search terms and a huge amount of clickstream data. We use this data to estimate organic traffic by looking at all the different search queries people make, the positions websites take for them in search results, and where people click. For the core updates, we decided to look at the traffic at the beginning of the Google core update and the traffic for 14 days afterwards. We have now allowed time for Google to make any changes in their various data centers. It also gives us time and allows our data to map the changes.

Our data is normalized to take the average for volumes over 12 months, which should largely offset seasonality. Elections are an exception, as they do not take place every year. We won't see any more recent coverage or search topics initially either, but should be able to include all popular searches and related clickstream data later.

We see a decrease in average traffic to the top media sites in each category over the period of a Google Core update from 2015 to the present.

Total traffic decrease conservative media: ‑2.65%
Total traffic decrease liberal media: ‑1.78%

These numbers are very close and not statistically significant when you consider that we are looking at traffic from 50 websites over a 6 year period. Up until the penultimate election in 2016, the effects for both categories were more or less the same. Until the last election in 2020, the effects were also more or less the same if you look at the results from the previous year, while conservative media benefited more from the last update.

If we take a closer look at each data point, we see that both conservative and liberal media sites saw positive and negative impacts during all Google Core updates. Each box plot in the diagram below represents the top websites in each category, and I repeat that every single update had winners and losers in both categories. Usually, whether a page wins or loses in a core update depends primarily on the quality of the page.

 

While we cannot conclude from this data that bias can be ruled out in Google's search results, we can say that no new bias has been introduced during Google Core updates within the last 6 years.

Is there a traffic bias?

One thing that stood out from our point of view is the fact that liberal sites definitely get more traffic than conservative sites.

The question that arises here is: why is that so? Does this indicate a potential bias predicated on our keyword data? Let's see if we can explain this difference in terms of traffic numbers.

Scope of content

If we look at the number of pages indexed, we find that liberal media sites have indexed more than eight times as many pages as conservative media sites. In fact, the graph is almost identical to the graph for the percentage of organic traffic. For this reason, liberal media sites generally have more chances of being ranked for different things than conservative media sites.

Branded vs. No Branded Traffic

Branded traffic is more or less the same for CNN and Fox News, which means a similar number of people search specifically for those media sites in organic search. However, branded traffic for CNN is a smaller percentage of total traffic, presumably because they simply have more content. CNN has indexed about 2.5 times more pages than Fox News, so they have a higher chance of ranking for different things.

About 40% of the traffic to CNN is branded traffic, which equates to about 32.5 million organic visits per month.

About 67% of Fox News traffic is branded traffic, which equates to about 33 million organic visits per month.

Different explanations for different traffic volumes

With only the help of Google search data, it is difficult to determine with certainty why liberal sites get more traffic than conservative sites. Conservatives may be making less use of Goolge. It is also possible that Conservatives receive news from TV, apps or social media rather than from Google. All of this may be true, but without additional data these statements cannot be confirmed and are therefore only guesses.

Other options that we could have checked for bias

If we looked at certain examples of search queries, I'm sure we would have found what would look like examples of bias. The problem with specific examples is that there is an inherent bias in the person doing the analysis. The data is complex and you would need good practice in determining whether the pages and content are relevant to specific searches. Unfortunately, this would be difficult and extremely subjective, which is why we decided against this approach.

We could also have included link data and links from other websites to the respective media pages. Again, I'm sure that this data would have indicated bias, as better-known and higher-ranking websites with more pages tend to naturally generate more links. We have already proven this in our backlink growth study.

We could have looked at specific examples of search terms displayed by Google through their autocomplete system, but we already know that many negative queries are removed from these results. Again, there is a fundamental bias and it would be difficult to prove a deliberate bias.

Final thoughts

There is always a fundamental bias. While our data shows that Google's core updates have not been biased on any particular site in recent years, we cannot with certainty rule out bias in other areas of Google's systems.

Do you have any questions about this data? Post them to me on Twitter.

Translated by Heike Radlanski. Heike deals with all aspects of online marketing and product management.