Duplicate Content and Search Engines: The Truth

Duplicate content has been an area that has received much debate over the past few years. Interactive Return always recommends to clients that duplicate content on multiple pages or domains should be avoided as a general rule. However, the question remains – does the Duplicate Content Penalty actually exist?  A recent article written by Google confirms that there are duplicate content issues that are necessary to be aware of for our search engine optimisation, but the mysterious Google “duplicate content penalty” is something that is highly misunderstood.

Google’s recent article aims to set the record straight on its Official Webmaster Blog, outlining how “there’s no such thing as a ‘duplicate content penalty’…not in the way most people mean when they say that.” The duplicate content penalty is something that has simply been misunderstood and misrepresented by and within the industry even by internationally renowned search engine marketers. While duplicate content remains a priority for consideration in our Search Engine Optimisation efforts, we must revise the way we think of duplicate content as something that we must take control of, rather than a penalty for which we can’t.

The Google blog outlines some penalties related to duplicate content, but states that for example if you’re “scraping content from other sites and republishing it, or if you republish content without adding any additional value,” your site may be filtered out because it would not be representing “unique” results for a search user. As Vishal Saxena outlines in his post on the issue, if plagiarism is taken so seriously in the “real world,” then there’s no reason why a search engine would like plagiarised work either.

Often considered a “no no” when it comes to the mythical duplicate content penalty is having multiple URLs on the same domain pointing to the same content. This is a great example of how the Duplicate content issue is misunderstood, as search engines do not “penalise” site owners that have this problem, but that this may affect your site performance.  People have been equating their lack of site performance with a duplicate content penalty on a search engine, when this is issue is in fact resulting from the fact that most content management systems, web sites, and search engines just don’t work well with multiple URL’s.

SEO Roundtable outlines that we have to watch out for is this non-malicious duplicate content, where we may have pages that are very similar to each other, like filtered down search results or product option pages, without realising that a search engine might be filtering it out as repetitive and therefore irrelevant. Google outlined the way they handed duplicate documents:

  • When we detect duplicate content, such as through variations caused by URL parameters, we group the duplicate URLs into one cluster.
  • We select what we think is the “best” URL to represent the cluster in search results.
  • We then consolidate properties of the URLs in the cluster, such as link popularity, to the representative URL.

What this means is that if we don’t take action, Google is the one in control of what is the “best” URL; and if you want to be in charge, they suggest including the preferred URL in your Sitemap.

One of the other worries for staying in control of our duplicate content is when the identical content appears on different websites, as outlined by Saxena at ArticleBase. He says that many of us may have submitted the same content to various social media, article submission or promotional websites with the intention of promoting our website; but unless we link back to the original content we may see the negative impact this may have.  So, always give credit where credit’s due, even if it’s to yourself!

Along the same lines, people may be concerned with syndicated content that may be feeding in from other sites or their own content going out; and whether this will penalise you if your content is copied on another site.  Straight from the Google Webmaster Help Page on Duplicate Content, we are told to “syndicate carefully.” If you are syndicating on other sites, Google will choose the version that is most appropriate for users in a given search; which might not be the version you prefer. But crucial to remember is that each site on which your content is syndicated must include a link back to the original article, or you might ask those who are using your syndicated material to “block the version on their sites with robots.txt.”

In summary, the “duplicate content penalty” as it is understood does not exist, but I would say that duplicate content should remain a key consideration when you are trying to stay in control of your search engine optimisation. Duplicate web pages and multiple URL’s are not “banned,” but they are be filtered out in order to ensure users the most relevant and unique search results.  So Interactive Return suggests that we don’t loose sleep over Duplicate Content, but certainly don’t forget how to control it to avoid it negatively affecting the overall performance of your site.

Emer Lawn, Online Marketing Executive, Interactive Return

6 Responses to “Duplicate Content and Search Engines: The Truth”

  1. Website Optimisation Sheffield (SEO) said:

    Sep 23, 08 at 1:40 pm

    I post articles as part of my optimisation campaign, and the difference between where the article is first posted is massive. By this I mean, if submitting an article on ‘bosch silver washing machines’ to 10 sites, the site where it is first cached by Google goes on to be high up the engines. The last site to be cached is usually not even listed in the results for that key phrases in the engines. Duplicate content is not the end of the world, but can be a waste of time if you are trying to optimise a page or site.

  2. How to buy domains » Blog Archive » Duplicate Content and Search Engines: The Truth said:

    Sep 23, 08 at 1:48 pm

    [...] Anthony Chambers wrote an interesting post today onHere’s a quick excerptA recent article written by Google confirms that there are duplicate content issues that are necessary to be aware of for our search engine optimisation, but the mysterious Google “duplicate content penalty” is something that is highly … [...]

  3. Emer Lawn said:

    Sep 23, 08 at 5:04 pm

    Thanks for your post! The interesting thing about your comment is that you are proving the fact that we do need to be aware of where we are posting our content, and by avoiding duplicate content; we may rank higher up in organic rankings. When you post to several sites, remember to always link back to the original souce (even if it is your own!!), because if you don’t; Google will only see one of the pages as relevant and it will become pointless for you to post to so many pages.

  4. Ruby said:

    Sep 24, 08 at 3:00 am

    Thank you for your informative post, I like the slant on how you wrote this post. I definitely agree that search engine optimization is important for websites nowadays. Be sure to update your blog with more of such informative posts!

  5. How Does Duplicate Content Effect Private Label Rights? said:

    Jun 01, 09 at 11:35 pm

    [...] Duplicate Content and Search Engines: The Truth » Online [...]

  6. Introspective said:

    Jul 13, 09 at 4:10 pm

    I used to publish my articles, but now I wander should I stop doing this, because the risk of duplicate content penalty. Should I stop publishing my articles on article directories?


Leave a Reply