API
The Gcore Customer Portal is being updated. Screenshots may not show the current version.
CDN
CDN
BillingCDN resources overviewOrigin groupPurgePrefetchReports
API
Chosen image
Home/CDN/Logs/Raw Logs (paid)

Raw Logs: export CDN resource logs to your storage

What is a Raw Logs feature?

Raw Logs is an option that enables an automatic export of CDN resource logs to your storage. Logs contain information about user requests sent to cache servers and pre-cache servers (if origin shielding is enabled).

Note: The feature is paid. To activate, contact us via support@gcore.com. After activation, enable "Raw Logs" in your control panel and configure export to S3, FTP, or SFTP storage.

What is a Raw Logs feature

What is the "Add logs from origin shielding" option?

If you are using the Origin Shielding feature, we recommend that you enable the "Add logs from origin shielding" option. This means that the report will include not only requests to cache services, but also those to the pre-cache server. As a result, you will receive more detailed information on resource usage.

Note: If your account does not have "Origin Shielding" switched on, this option will not be available when setting up Raw logs.  

To enable "Add logs from origin shielding", tick the appropriate box when setting up Raw Logs (step #2 in the guide below).

Add logs from origin shielding

How is traffic calculated in log reports?

Logs can generate various types of analytics, such as traffic delivered. To understand what the totals mean, we recommend that you familiarize yourself with the formulas for calculating the logs.

The formula for calculating traffic depends on whether you use the "Add logs from origin shielding" option.

  1. If this option is disabled, the formula will look like this:
total_bytes = upstream_bytes + sent_bytes 

Where:

  • upstream_bytes are equal to the $upstream_response_length log field and refer to the response length from an origin in bytes
  • sent_bytes are equal to the $bytes_sent log field and refer to the number of bytes sent to a user from the edge (cache) servers

For example, if $upstream_response_length is 10485760 (bytes) and $bytes_sent is 1514848 (bytes), the final value in the Raw logs report will be 12,000,608 (bytes).

  1. If the "Add logs from origin shielding" option is enabled, the formula will look like this:
total_bytes = upstream_bytes + sent_bytes + shield_bytes

Where:

  • upstream_bytes are equal to the $upstream_response_length log field and refer to the response length from an origin in bytes
  • sent_bytes are equal to the $bytes_sent log field and refer to the number of bytes sent to a user from the edge (cache) servers
  • shield_bytes are equal to the $bytes_sent log field and refer to the number of bytes sent to a user from the pre-cache server

Note: The final value of log data may differ slightly from the billing statistics, as there may be cases where not all logs are received, such as:

  • You are using the "Origin Shielding" feature but did not check the "Add logs from origin shielding" box
  • You have a rate limit set on your storage side, and when CDN started generating logs, some logs were not downloaded because of the rate limitation

Export logs to an S3 storage

Amazon storage

1. Leave the box "Do not send empty logs" checked if you don't want to receive empty logs. If you want to receive empty logs, uncheck it.

2. If you use our origin shielding option, you can see a checkbox "Add logs from origin shielding". Check the box if you want to receive logs from both edge servers and pre-cache shielding servers.

3. For storage type, select "Amazon".

4. Specify your access key ID. In your Amazon personal account, it is called "AWS access Key ID". You can find it using the instruction. An access key ID and secret access key are required to configure log export to your storage.

5. Specify your secret access key. In your Amazon personal account, it is called "AWS secret access Key". You can find it using the instruction.

6. Specify your AWS region — the location of a server where your storage is hosted. This is optional: for most storages, the region is determined automatically. You can leave the field empty. But we recommend filling it out to ensure that your logs are exported successfully.

7. Choose how to organize storage: put logs of all CDN resources into one bucket or use separate buckets for each CDN resource.

8. Specify bucket(s) for log export. Make sure you indicate an existing bucket. Otherwise, your logs cannot be exported. Specify a folder name if you want to export logs to a specific folder within a bucket.

9. Click Save changes.

Raw logs

Non-Amazon storage

1. Leave the box "Do not send empty logs" checked if you don't want to receive empty logs. If you want to receive empty logs, uncheck it.

2. If you use our origin shielding option, you can see a checkbox "Add logs from origin shielding". Check the box if you want to receive logs from both edge servers and pre-cache shielding servers.

3. For storage type, select "Other".

4. Specify a hostname — a name that is assigned to a storage server within a network and is used instead of an IP address. If you use Gcore S3 storage, you can find its access key ID in your personal account in the "Hostname" field.

5. Specify your access key ID. Along with a secret access key, it is required to configure log export to your storage. If you use Gcore S3 storage, you can find its access key ID in your personal account in the "Access key" field.

6. Specify your secret access key. If you use Gcore S3 storage, you can find its secret access key in your personal account in the "Secret key" field.

7. Specify a bucket hostname — a bucket ID that is used by your S3 storage system in the {bucket_name}.{hostname} format. It is required to ensure that logs are exported to a correct bucket within a storage. A bucket hostname of the Gcore storage looks as follows: {bucket name}.{hostname from step 3}. For example: examplename.s-ed1.cloud.gcore.lu.

8. Specify a region — location ID of a server where your storage is hosted. This is optional: for some storages, the region is determined automatically. You can leave the field empty. If you use Gcore S3 storage, a location ID is required. You can find it in the "Details" of the storage. Your location ID is a part of your hostname to the first dot.

Specify a region

9. Choose how to organize storage: put logs of all CDN resources into one bucket or to use separate buckets for each CDN resource.

10. Specify bucket(s) for log export. Make sure you indicate an existing bucket. Otherwise, your logs cannot be exported. If you want to export logs to a specific folder within a bucket, specify a folder name.

11. Click Save changes.

Specify a region

Export logs to an FTP/SFTP storage

1. Leave the box "Do not send empty logs" checked if you don't want to receive empty logs. If you want to receive empty logs, uncheck it.

2. If you use our origin shielding option, you can see a checkbox "Add logs from origin shielding". Check the box if you want to receive logs from both edge servers and pre-cache shielding servers.

3. Specify a hostname — a name that is assigned to a storage server within a network and is used instead of an IP address. If you use Gcore SFTP storage, you can find its hostname in the "Details" of the storage in the "Hostname" field. It looks as follows: ams.origin.gcdn.co. Additionally, you can specify an FTP or SFTP storage port by adding a colon after the hostname. For example: ams.origin.gcore.co:2200.

4. Specify a storage username. If you use Gcore SFTP storage, you can find its username in the "Details" of your storage in the "Storage/User name" field.

5. Enter your storage password.

6. Specify a folder for export. If you use Gcore SFTP storage, specify the root (home) folder where other folders originate from. You can find its name in the "Details" of your SFTP storage at the end of the "Upload path" field.

Raw logs

If you use an SFTP storage from another provider, clarify whether a root folder that includes other folders is created by default. If not, leave the field empty. If yes, specify a folder name.

7. Choose how to organize storage: put logs of all CDN resources into one folder or to use separate folders for each CDN resource. Then specify a folder name. If you specify a non-existent folder, logs will be exported to a root folder.

8. Click Save changes.

Specify a folder for export

Export time intervals

Logs are exported at the end of each hour. If you activate Raw Logs at 00:30, the first logs will be exported between 00:45 and 01:00, and the next ones — between 01:45 and 02:00.

If CDN servers are not requested and the box "Do not send empty logs" is unchecked, an empty log file (± 20 bytes) will be sent to your storage.

You can see the status of the Raw Logs option in your control panel:

  • "Pending" is a status for the time interval between the connection to a storage and the very first log export
  • "OK" is a status showing that logs are exported from at least one CDN server
  • "Failed" is a status indicating that an error occurred while connecting to a storage or that the service failed to export logs within 24 hours
  • "Pause" is a status showing that the option is paused
Raw logs

Log path example

s3://log-bucket-name/2019/08/20/15/nodename_primarycname.domain.ru_access.log.gz

Log format

"$remote_addr" "-" "$remote_user" "[$time_local]" "$request" "$status"  
"$body_bytes_sent" "$http_referer" "$http_user_agent" "$bytes_sent"  
"$edgename" "$scheme" "$host" "$request_time"  
"$upstream_response_time" "$request_length" "$http_range" "[$responding_node]"  
"$upstream_cache_status" "$upstream_response_length" "$upstream_addr"  
"$gcdn_api_client_id" "$gcdn_api_resource_id" "$uid_got" "$uid_set"  
"$geoip_country_code" "$geoip_city" "$shield_type" "$server_addr" "$server_port"  
"$upstream_status" "-" "$upstream_connect_time" "$upstream_header_time"  
"$shard_addr" "$geoip2_data_asnumber" "$connection" "$connection_requests"  
"$http_traceparent" "$http_x_forwarded_proto" "$gcdn_internal_status_code" "$ssl_cipher"  
"$ssl_session_id" "$ssl_session_reused" "$sent_http_content_type" "$tcpinfo_rtt" 
"$server_country_code" "$gcdn_tcpinfo_snd_cwnd" "$gcdn_tcpinfo_total_retrans" "$gcdn_rule_id" 

Please don’t be surprised if you see a field that is not listed above. We occasionally add new fields. If some fields are added to logs, you will receive an email about it. New fields are added to the end of the line.

Log example

"0.0.0.0" "-" "-" "[26/Apr/2019:09:47:40 +0000]" "GET /ContentCommon/images/image.png HTTP/1.1"  
"200" "1514283" "https://example.com/videos/10" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1)  
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 YaBrowser/16.10.0.2309 Safari/537.36"  
"1514848" "[dh-up-gc18]" "https" "origin.cdn.com" "1.500" "0.445" "157" "bytes=0-1901653" "[dh]"  
"MISS" "10485760" "0.0.0.0:80" "2510" "7399" "-" "-" "KZ" "-" "shield_no" "0.0.0.0" "80" "206" "-" "0.000"  
"0.200" "0.0.0.0" "asnumber" "106980391" "1" "00-d5fe1dc9035165ce36952daf29686b6c-14330be33197dd1a-01" "-" "-"  
"ECDHE-RSA-AES256-GCM-SHA384" "28a4184139cb43cdc79006cf2d1a4ac93bdc****" "r"  
"application/json" "21" "PL" "45" "10" "100700"

Log fields

Not all fields are important. Some of them relate to our internal CDN system and are not meaningful for you. In the table below, we have highlighted such system fields in italics. Other fields can be helpful for traffic analysis or statistics.

Field Log value example Description
$remote_addr 0.0.0.0 User's IP address
$remote_user
(internal system variable)
- Username used in Basic authentication
[$time_local] [26/Apr/2019
:09:47:40 +0000]
Local time in Common Log Format
$request GET /ContentCommon/
images/image.png
HTTP/1.1
HTTP method, requested file path, HTTP version
$status 200 Response status code from a CDN server
$body_bytes_sent 1514283 Number of bytes sent to a user, excluding the response header size
$http_referer https://example.com
/videos/10
Referrer - a URL requested by a user
$http_user_agent Mozilla/5.0
(Macintosh; Intel
Mac OS X 10_12_1)
AppleWebKit/537.36
(KHTML, like Gecko)
Chrome/53.0.2785.116
YaBrowser/16.10.0.2309
Safari/537.36
User agent that was used to send a request (browser or other application)
$bytes_sent 1514848 Number of bytes sent to a user
$edgename [dh-up-gc18] CDN server that forwarded a requested file
$scheme https Protocol (HTTP or HTTPS) of a request
$host cdn.example.com Requested hostname of a CDN resource
$request_time 1.500 Request processing time in seconds (accurate to milliseconds); time elapsed between the first bytes of a request were processed and logging after the last bytes were sent to a user
$upstream_response_time 0.445 Number of seconds (accurate to milliseconds) it took to receive a response from an origin. In case of multiple responses, commas and colons are used
$request_length 157 Request length (including request line, header, and request body)
$http_range bytes=0-1901653 File fragment size in a Range request
[$responding_node] dh Responding data center
$upstream_cache_status MISS Status of a requested file in CDN cache:
- HIT is a status of a response served from CDN cache.
- STALE is a status of an outdated response that failed to update because an origin was not responding or responding incorrectly.
- UPDATING is a status of an outdated response that is still updating since a previous request.
- REVALIDATED is a status of a response that is identical to the one on an origin based on the proxy_cache_revalidate directive.
- EXPIRED is a status of a response that has expired in cache, but still matches the one on an origin. A request has been sent to an origin for re-caching.
- MISS is a status of a response that has been served directly from an origin, rather than from cache.
- BYPASS is a status of a response for the first file request after clearing the cache.
Note: this status appears when the file is requested by each CDN server.
When one CDN server requests a file for the first time, it will have the BYPASS status.
When the same server requests the file again, the status will be changed to HIT.
When another CDN server requests the file, it will again have the BYPASS status.
$upstream_response_length 10485760 Response length from an origin in bytes. In case of multiple responses, commas and colons are used
$upstream_addr 0.0.0.0:80 Origin's IP address and port
$gcdn_api_client_id
(internal system variable)
123 Your ID in our system
$gcdn_api_resource_id
(internal system variable)
01 Your CDN-resource ID in our system
$uid_got
(internal system variable)
- Cookie name and received user ID
$uid_set
(internal system variable)
- Cookie name and provided user ID
$geoip_country_code KZ User’s country code according to the ISO 3166 standard (Alpha-2 code).
$geoip_city - User’s city code
$shield_type
(internal system variable)
shield_no This field shows whether the shielding option is enabled:
shield_old - enabled
shield_no - disabled
$server_addr
(internal system variable)
0.0.0.0 IP address of an Anycast zone or CDN server
$server_port
(internal system variable)
80 Requested port
$upstream_status 206 Origin response code
$upstream_connect_time 0.000 Number of seconds (accurate to milliseconds) it took to access an origin server
$upstream_header_time 0.200 Number of seconds (accurate to milliseconds) it took to receive a response header from an origin server
$shard_addr
(internal system variable)
0.0.0.0 IP address of a CDN server that was first to accept a request if the Cache Sharding feature is enabled
$geoip2_data_asnumber asnumber Number of an autonomous system that sent a request
$connection
(internal system variable)
2897494295 Connection serial number
$connection_requests
(internal system variable)
1 Current number of requests made through a connection
$http_traceparent
(internal system variable)
00-d5fe1dc903
5165ce36952da
f29686b6c-143
30be33197dd1a
-01
Unique request identifier, more info in the Traceparent guide
$http_x_forwarded_proto - Initial protocol of an incoming request (HTTP or HTTPS)
$gcdn_internal_status_code,
(internal system variables)
- Initial status code. Possible values are: -, or 100700
$ssl_cipher
(internal system variable)
ECDHE-RSA-AES256
-GCM-SHA384
Cipher name used for an established SSL connection
$ssl_session_id
(internal system variable)
28a4184139cb43
cdc79006cf2d1
a4ac93bdc****
Session ID of an established SSL connection
$ssl_session_reused
(internal system variable)
r The filed shows whether a session was reused (“r” ) or not (“.”)
$sent_http_content_type application/json Value of the Content-Type HTTP header, indicating the MIME type of a transmitted file
$tcpinfo_rtt 21 Average time (latency) it takes to transfer a packet to/from a server. The unit of time is microseconds.
$server_country_code PL Server’s country code according to the ISO 3166 standard (Alpha-2 code).
$gcdn_tcpinfo_snd_cwnd 45 Size of the TCP Congestion window, i.e. the maximum number of TCP segments that the connection can send before an acknowledgement is required.
$tcpi_total_retrans 10 Total number of retransmitted packets over the life of the connection.
$gcdn_rule_id 100700 Initial rule ID (beta). Possible values are: -, or 100700

Was this article helpful?

Not a Gcore user yet?

Learn more about our next-gen CDN

Go to the product page