Avoid indexing of your CDN resource by web crawlers

Search engines use web crawlers to index content from various websites, making it discoverable to users. However, when your CDN resources are indexed, it can lead to problems, so we’ll explain how to avoid it here.

What is the problem with CDN custom domain indexing?

When you create a CDN resource for static assets for your original website, part of your content is available from the main (e.g., website-example.com/image.png) and CDN custom domains (e.g., cdn.website-example.com/image.png.) This can lead to duplicated content in search results which negatively affects SEO, degrading your site’s position on search engines like Google. By using the Status code option, you can forbid crawlers from indexing custom domains hence avoiding duplicate content.

How to prohibit web crawlers from indexing your CDN custom domain

1. Go to the CDN resource and click the custom domain of the CDN resource you want to prohibit from indexing.

A new page will open, where you will perform the remaining steps.

2. Click Content, then Status code.

3. Enable the Status code option.

4. In the URL or code text (optional) field, enter the following directive:

      Copy
      
    User-agent: *\nDisallow: /\n

5. Save changes.

That’s it. You have successfully prevented content requested from the custom CDN domain from being indexed.

Note: This configuration doesn’t affect the website itself.