Raw Logs: export CDN resource logs to your storage

What is a Raw Logs feature?

Raw Logs is an option that enables an automatic export of CDN resource logs to your storage. Logs contain information about user requests sent to cache servers and pre-cache servers (if origin shielding is enabled).

Note: The feature is paid. To activate, contact us via support@gcore.com. After activation, enable "Raw Logs" in your control panel and configure export to S3, FTP, or SFTP storage.

What is the "Add logs from origin shielding" option?

If you are using the Origin Shielding feature, we recommend that you enable the "Add logs from rigin shielding" option. This means that the report will include not only requests to cache services, but also those to the pre-cache server. As a result, you will receive more detailed information on resource usage.

Note: If your account does not have "Origin Shielding" switched on, this option will not be available when setting up Raw logs.  

To enable "Add logs from origin shielding", tick the appropriate box when setting up Raw Logs (step #2 in the guide below).

How is traffic calculated in log reports?

Logs can generate various types of analytics, such as traffic delivered. To understand what the totals mean, we recommend that you familiarize yourself with the formulas for calculating the logs.

The formula for calculating traffic depends on whether you use the "Add logs from origin shielding" option.

1. If this option is disabled, the formula will look like this:

      Copy
      
    total_bytes = upstream_bytes + sent_bytes 

Where:

• upstream_bytes are equal to the $upstream_response_length log field and refer to the response length from an origin in bytes
• sent_bytes are equal to the $bytes_sent log field and refer to the number of bytes sent to a user from the edge (cache) servers

For example, if $upstream_response_length is 10485760 (bytes) and $bytes_sent is 1514848 (bytes), the final value in the Raw logs report will be 12,000,608 (bytes).

2. If the "Add logs from origin shielding" option is enabled, the formula will look like this:

      Copy
      
    total_bytes = upstream_bytes + sent_bytes + shield_bytes

Where:

• upstream_bytes are equal to the $upstream_response_length log field and refer to the response length from an origin in bytes
• sent_bytes are equal to the $bytes_sent log field and refer to the number of bytes sent to a user from the edge (cache) servers
• shield_bytes are equal to the $bytes_sent log field and refer to the number of bytes sent to a user from the pre-cache server

Note: The final value of log data may differ slightly from the billing statistics, as there may be cases where not all logs are received, such as:

• You are using the "Origin Shielding" feature but did not check the "Add logs from origin shielding" box
• You have a rate limit set on your storage side, and when CDN started generating logs, some logs were not downloaded because of the rate limitation

Export logs to an S3 storage

Amazon storage

1. Leave the box "Do not send empty logs" checked if you don't want to receive empty logs. If you want to receive empty logs, uncheck it.

2. If you use our origin shielding option, you can see a checkbox "Add logs from origin shielding". Check the box if you want to receive logs from both edge servers and pre-cache shielding servers.

3. For storage type, select "Amazon".

4. Specify your access key ID. In your Amazon personal account, it is called "AWS access Key ID". You can find it using the instruction. An access key ID and secret access key are required to configure log export to your storage.

5. Specify your secret access key. In your Amazon personal account, it is called "AWS secret access Key". You can find it using the instruction.

6. Specify your AWS region — the location of a server where your storage is hosted. This is optional: for most storages, the region is determined automatically. You can leave the field empty. But we recommend filling it out to ensure that your logs are exported successfully.

7. Choose how to organize storage: put logs of all CDN resources into one bucket or use separate buckets for each CDN resource.

8. Specify bucket(s) for log export. Make sure you indicate an existing bucket. Otherwise, your logs cannot be exported. Specify a folder name if you want to export logs to a specific folder within a bucket.

9. Click Save changes.

Non-Amazon storage

1. Leave the box "Do not send empty logs" checked if you don't want to receive empty logs. If you want to receive empty logs, uncheck it.

2. If you use our origin shielding option, you can see a checkbox "Add logs from origin shielding". Check the box if you want to receive logs from both edge servers and pre-cache shielding servers.

3. For storage type, select "Other".

4. Specify a hostname — a name that is assigned to a storage server within a network and is used instead of an IP address. If you use Gcore S3 storage, you can find its access key ID in your personal account in the "Hostname" field.

5. Specify your access key ID. Along with a secret access key, it is required to configure log export to your storage. If you use Gcore S3 storage, you can find its access key ID in your personal account in the "Access key" field.

6. Specify your secret access key. If you use Gcore S3 storage, you can find its secret access key in your personal account in the "Secret key" field.

7. Specify a bucket hostname — a bucket ID that is used by your S3 storage system in the {bucket_name}.{hostname} format. It is required to ensure that logs are exported to a correct bucket within a storage. This field is optional: for some storages, a bucket hostname is determined automatically. If you use Gcore or Yandex.Cloud storage, a bucket hostname is required. A bucket hostname of the Gcore storage looks as follows: {bucket name}.{hostname from step 3}. For example: examplename.s-ed1.cloud.gcore.lu. A bucket hostname of Yandex.Cloud storage looks as follows: {bucket name}.{Yandex.Cloud hostname}. For example: examplename.storage.yandexcloud.net.

8. Specify a region — location ID of a server where your storage is hosted. This is optional: for some storages, the region is determined automatically. You can leave the field empty. If you use Gcore S3 storage, a location ID is required. You can find it in the "Details" of the storage. Your location ID is a part of your hostname to the first dot.

9. Choose how to organize storage: put logs of all CDN resources into one bucket or to use separate buckets for each CDN resource.

10. Specify bucket(s) for log export. Make sure you indicate an existing bucket. Otherwise, your logs cannot be exported. If you want to export logs to a specific folder within a bucket, specify a folder name.

11. Click Save changes.

Export logs to an FTP/SFTP storage

1. Leave the box "Do not send empty logs" checked if you don't want to receive empty logs. If you want to receive empty logs, uncheck it.

2. If you use our origin shielding option, you can see a checkbox "Add logs from origin shielding". Check the box if you want to receive logs from both edge servers and pre-cache shielding servers.

3. Specify a hostname — a name that is assigned to a storage server within a network and is used instead of an IP address. If you use Gcore SFTP storage, you can find its hostname in the "Details" of the storage in the "Hostname" field. It looks as follows: ams.origin.gcdn.co. Additionally, you can specify an FTP or SFTP storage port by adding a colon after the hostname. For example: ams.origin.gcore.co:2200.

4. Specify a storage username. If you use Gcore SFTP storage, you can find its username in the "Details" of your storage in the "Storage/User name" field.

5. Enter your storage password.

6. Specify a folder for export. If you use Gcore SFTP storage, specify the root (home) folder where other folders originate from. You can find its name in the "Details" of your SFTP storage at the end of the "Upload path" field.

If you use an SFTP storage from another provider, clarify whether a root folder that includes other folders is created by default. If not, leave the field empty. If yes, specify a folder name.

7. Choose how to organize storage: put logs of all CDN resources into one folder or to use separate folders for each CDN resource. Then specify a folder name. If you specify a non-existent folder, logs will be exported to a root folder.

8. Click Save changes.

Export time intervals

Logs are exported at the end of each hour. If you activate Raw Logs at 00:30, the first logs will be exported between 00:45 and 01:00, and the next ones — between 01:45 and 02:00.

If CDN servers are not requested and the box "Do not send empty logs" is unchecked, an empty log file (± 20 bytes) will be sent to your storage.

You can see the status of the Raw Logs option in your control panel:

• "Pending" is a status for the time interval between the connection to a storage and the very first log export
• "OK" is a status showing that logs are exported from at least one CDN server
• "Failed" is a status indicating that an error occurred while connecting to a storage or that the service failed to export logs within 24 hours
• "Pause" is a status showing that the option is paused

Log path example

      Copy
      
    s3://log-bucket-name/2019/08/20/15/nodename_primarycname.domain.ru_access.log.gz

Log format

      Copy
      
    "$remote_addr" "-" "$remote_user" "[$time_local]" "$request" "$status"  
"$body_bytes_sent" "$http_referer" "$http_user_agent" "$bytes_sent"  
"$edgename" "$scheme" "$host" "$request_time"  
"$upstream_response_time" "$request_length" "$http_range" "[$responding_node]"  
"$upstream_cache_status" "$upstream_response_length" "$upstream_addr"  
"$gcdn_api_client_id" "$gcdn_api_resource_id" "$uid_got" "$uid_set"  
"$geoip_country_code" "$geoip_city" "$shield_type" "$server_addr" "$server_port"  
"$upstream_status" "-" "$upstream_connect_time" "$upstream_header_time"  
"$shard_addr" "$geoip2_data_asnumber" "$connection" "$connection_requests"  
"$request_id" "$http_x_forwarded_proto" "$http_x_forwarded_request_id" "$ssl_cipher"  
"$ssl_session_id" "$ssl_session_reused"  
"$sent_http_content_type" "$tcpinfo_rtt" 

Please don’t be surprised if you see a field that is not listed above. We occasionally add new fields. If some fields are added to logs, you will receive an email about it. New fields are added to the end of the line.

Log example

      Copy
      
    "0.0.0.0" "-" "-" "[26/Apr/2019:09:47:40 +0000]" "GET /ContentCommon/images/image.png HTTP/1.1"  
"200" "1514283" "https://example.com/videos/10" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1)  
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 YaBrowser/16.10.0.2309 Safari/537.36"  
"1514848" "[dh-up-gc18]" "https" "origin.cdn.com" "1.500" "0.445" "157" "bytes=0-1901653" "[dh]"  
"MISS" "10485760" "0.0.0.0:80" "2510" "7399" "-" "-" "KZ" "-" "shield_no" "0.0.0.0" "80" "206" "-" "0.000"  
"0.200" "0.0.0.0" "asnumber" "106980391" "1" "c1c0f12ab35b7cccccd5dc0a454879c5" "-" "-"  
"ECDHE-RSA-AES256-GCM-SHA384" "28a4184139cb43cdc79006cf2d1a4ac93bdc****" "r"  
"application/json" "21"

Log fields

Not all fields are important. Some of them relate to our internal CDN system and are not meaningful for you. In the table below, we have highlighted such system fields in italics. Other fields can be helpful for traffic analysis or statistics.