Skip to main content

Documentation Index

Fetch the complete documentation index at: https://gcore.com/docs/llms.txt

Use this file to discover all available pages before exploring further.

Search engines use web crawlers to index content from various websites, making it discoverable to users. However, when your CDN resources are indexed, it can lead to problems, so we’ll explain how to avoid it here.

​
What is the problem with CDN custom domain indexing?

When you create a CDN resource for static assets for your original website, part of your content is available from the main (e.g., website-example.com/image.png) and CDN custom domains (e.g., cdn.website-example.com/image.png.) This can lead to duplicated content in search results which negatively affects SEO, degrading your site’s position on search engines like Google. By using the Status code option, you can forbid crawlers from indexing custom domains hence avoiding duplicate content.

​
How to prohibit web crawlers from indexing your CDN custom domain

1. Go to the CDN resource and click the custom domain of the CDN resource you want to prohibit from indexing.
CDN resource
A new page will open, where you will perform the remaining steps.
A new page
2. Click Content, then Status code. 3. Enable the Status code option. 4. In the URL or code text (optional) field, enter the following directive: 
User-agent: *\nDisallow: /\n
5. Save changes. That’s it. You have successfully prevented content requested from the custom CDN domain from being indexed. Note : This configuration doesn’t affect the website itself.