API
The Gcore Customer Portal is being updated. Screenshots may not show the current version.
CDN
CDN
BillingCDN resources overviewOrigin groupPurgePrefetchReports
API
Chosen image
Home/CDN

Logs uploader (near real-time)

Logs Uploader is a feature that enables an automatic export of CDN resource logs to your storage in near real-time.

Exported logs contain information about user requests sent to cache servers and pre-cache servers (if you have the Origin shielding feature enabled).

This is a paid feature. To activate Logs Uploader, contact the Gcore support team.

Logs uploader settings

In this section, you can find general information about log settings, statuses, and how to configure logs exporting for different storage types.

Log format example

"$remote_addr" "-" "$remote_user" "[$time_local]" "$request" "$status"  
"$body_bytes_sent" "$http_referer" "$http_user_agent" "$bytes_sent"  
"$edgename" "$scheme" "$host" "$request_time"  
"$upstream_response_time" "$request_length" "$http_range" "[$responding_node]"  
"$upstream_cache_status" "$upstream_response_length" "$upstream_addr"  
"$gcdn_api_client_id" "$gcdn_api_resource_id" "$uid_got" "$uid_set"  
"$geoip_country_code" "$geoip_city" "$shield_type" "$server_addr" "$server_port"  
"$upstream_status" "-" "$upstream_connect_time" "$upstream_header_time"  
"$shard_addr" "$geoip2_data_asnumber" "$connection" "$connection_requests"  
"$http_traceparent" "$http_x_forwarded_proto" "$gcdn_internal_status_code" "$ssl_cipher"  
"$ssl_session_id" "$ssl_session_reused" "$sent_http_content_type" "$tcpinfo_rtt" 
"$server_country_code" "$gcdn_tcpinfo_snd_cwnd" "$gcdn_tcpinfo_total_retrans" "$gcdn_rule_id" 

It’s OK if you find a field that’s not listed in the example. We occasionally add new fields to the end of the line. If some fields are added to logs, you’ll receive an email about the update.

Log fields

The following table contains a complete list of available log fields. Fields formatted in italics relate to our internal CDN system, so you can ignore them.

You can check other fields—they can be helpful for traffic analysis or statistics.

Field Log value example Description
$remote_addr 0.0.0.0 User's IP address
$remote_user
(internal system variable)
- Username used in Basic authentication
[$time_local] [26/Apr/2019
:09:47:40 +0000]
Local time in Common Log Format
$request GET /ContentCommon/
images/image.png
HTTP/1.1
HTTP method, requested file path, HTTP version
$status 200 Response status code from a CDN server
$body_bytes_sent 1514283 Number of bytes sent to a user, excluding the response header size
$http_referer https://example.com
/videos/10
Referrer - a URL requested by a user
$http_user_agent Mozilla/5.0
(Macintosh; Intel
Mac OS X 10_12_1)
AppleWebKit/537.36
(KHTML, like Gecko)
Chrome/53.0.2785.116
YaBrowser/16.10.0.2309
Safari/537.36
User agent that was used to send a request (browser or other application)
$bytes_sent 1514848 Number of bytes sent to a user
$edgename [dh-up-gc18] CDN server that forwarded a requested file
$scheme https Protocol (HTTP or HTTPS) of a request
$host cdn.example.com Requested hostname of a CDN resource
$request_time 1.500 Request processing time in seconds (accurate to milliseconds); time elapsed between the first bytes of a request were processed and logging after the last bytes were sent to a user
$upstream_response_time 0.445 Number of seconds (accurate to milliseconds) it took to receive a response from an origin. In case of multiple responses, commas and colons are used
$request_length 157 Request length (including request line, header, and request body)
$http_range bytes=0-1901653 File fragment size in a Range request
[$responding_node] dh Responding data center
$upstream_cache_status MISS Status of a requested file in CDN cache:
HIT is the status of a response served from the CDN cache.
STALE is the status of an outdated response that failed to update because an origin was not responding or responding incorrectly.
UPDATING is the status of an outdated response that is still updating since a previous request.
REVALIDATED is the status of a response that is identical to the one on an origin based on the proxy_cache_revalidate directive.
EXPIRED is the status of a response that has expired in cache but still matches the one on an origin. A request has been sent to an origin for re-caching.
MISS is the status of a response that has been served directly from an origin rather than from a cache.
BYPASS is the status of a response for the first file request after clearing the cache. This status appears when the file is requested by each CDN server.
When one CDN server requests a file for the first time, it will have the BYPASS status.
When the same server requests the file again, the status will be changed to HIT.
When another CDN server requests the file, it will again have the BYPASS status.
$upstream_response_length 10485760 Response length from an origin in bytes. In case of multiple responses, commas and colons are used
$upstream_addr 0.0.0.0:80 Origin's IP address and port
$gcdn_api_client_id
(internal system variable)
123 Your ID in our system
$gcdn_api_resource_id
(internal system variable)
01 Your CDN-resource ID in our system
$uid_got
(internal system variable)
- Cookie name and received user ID
$uid_set
(internal system variable)
- Cookie name and provided user ID
$geoip_country_code KZ User’s country code according to the ISO 3166 standard (Alpha-2 code).
$geoip_city - User’s city code
$shield_type
(internal system variable)
shield_no This field shows whether the shielding option is enabled:
shield_old - enabled
shield_no - disabled
$server_addr
(internal system variable)
0.0.0.0 IP address of an Anycast zone or CDN server
$server_port
(internal system variable)
80 Requested port
$upstream_status 206 Origin response code
$upstream_connect_time 0.000 Number of seconds (accurate to milliseconds) it took to access an origin server
$upstream_header_time 0.200 Number of seconds (accurate to milliseconds) it took to receive a response header from an origin server
$shard_addr
(internal system variable)
0.0.0.0 IP address of a CDN server that was first to accept a request if the Cache Sharding feature is enabled
$geoip2_data_asnumber asnumber Number of an autonomous system that sent a request
$connection
(internal system variable)
2897494295 Connection serial number
$connection_requests
(internal system variable)
1 Current number of requests made through a connection
$http_traceparent
(internal system variable)
00-d5fe1dc903
5165ce36952da
f29686b6c-143
30be33197dd1a
-01
Unique request identifier, more info in the Traceparent guide
$http_x_forwarded_proto - Initial protocol of an incoming request (HTTP or HTTPS)
$gcdn_internal_status_code,
(internal system variables)
- Initial status code. Possible values are: -, or 100700
$ssl_cipher
(internal system variable)
ECDHE-RSA-AES256
-GCM-SHA384
Cipher name used for an established SSL connection
$ssl_session_id
(internal system variable)
28a4184139cb43
cdc79006cf2d1
a4ac93bdc****
Session ID of an established SSL connection
$ssl_session_reused
(internal system variable)
r The field shows whether a session was reused (“r” ) or not (“.”)
$sent_http_content_type application/json Value of the Content-Type HTTP header, indicating the MIME type of a transmitted file
$tcpinfo_rtt 21 Average time (latency) it takes to transfer a packet to/from a server. The unit of time is microseconds.
$server_country_code PL Server’s country code according to the ISO 3166 standard (Alpha-2 code).
$gcdn_tcpinfo_snd_cwnd 45 Size of the TCP Congestion window, i.e., the maximum number of TCP segments that the connection can send before an acknowledgment is required.
$tcpi_total_retrans 10 Total number of retransmitted packets over the life of the connection.
$gcdn_rule_id 100700 Initial rule ID (beta). Possible values are: -, or 100700

Configure logs for export

You can enable and set up the Logs uploader feature in the Gcore Customer Portal on the Gcore CDN page.

To access log settings, navigate to Logs > Logs uploader and configure the feature as described in the following steps.

Step 1 (optional). Include empty logs

Keep the Do not send empty logs option selected if you don't want to receive empty logs. Otherwise, uncheck it.

Step 2 (optional). Enable origin shielding

If you are using the origin shielding feature, you’ll see the Add logs from origin shielding checkbox when configuring Logs uploader.

We recommend that you select this option as it ensures that the logs report will include both requests to cache services and requests to the pre-cache server. Thus, you’ll receive more detailed information on resource usage. To include these logs, select the checkbox to enable Include logs from origin shielding.

Include logs from origin shielding checkbox highlighted

If you don’t see the origin shielding option on the Logs uploader page, this feature is not activated for your account. For details on how to activate origin shielding, check our dedicated guide.

Step 3. Select log fields

Choose which log fields you want to include in the exported report. By default, all fields are selected.

Step 4. Configure a storage provider

Amazon

Follow these instructions to export logs to AWS storage:

1. In the Storage provider, select Amazon.

2. Provide your access key ID and secret access key, which together form long-term AWS credentials.

3. (Optional). Choose a storage region. While the region is often determined automatically, we recommend specifying it to ensure that your logs are exported successfully.

4. Specify the name of a bucket where you want to export CDN logs.

5. (Optional). Enter a folder name if you want to export logs to a specific folder within a bucket.

Receive logs to AWS

6. Click Save changes to apply the updates.

OSS

Follow these instructions to export logs to configure logs for Alibaba Cloud Object Storage Service (OSS):

1. In the Storage provider, select OSS.

2. Provide your access key ID and secret access key. Check the official OSS documentation for details.

3. (Optional). Choose a storage region. While the region is often determined automatically, we recommend specifying it to ensure that your logs are exported successfully.

4. Specify the name of a bucket where you want to export CDN logs.

5. (Optional). Enter a folder name if you want to export logs to a specific folder within a bucket.

Receive logs to OSS

6. Click Save changes to apply the updates.

Gcore\Other

Follow these instructions to export logs to Gcore Object Storage or any S3 storage of your choice:

1.In the Storage provider, select Gcore or Other.

2. Specify a hostname—a name that’s assigned to a storage server within a network and is used instead of an IP address. In you’re using Gcore Storage, you can find your hostname in the storage details section.

3. Provide your access key ID and secret access key. You can find this information in the Details of the storage.

4. Specify the name of a bucket where you want to export CDN logs.

5. (Optional). Enter a folder name if you want to export logs to a specific folder within a bucket.

Receive logs to AWS

6. Click Save changes to apply the updates.

How near real-time log exporting works

Logs are achived and exported to your endpoint every five minutes. If CDN servers are not requested and you didn’t select the Do not send empty logs checkbox when configuring Logs uploader, an empty log file (± 20 bytes) will be sent to your storage.

Was this article helpful?

Not a Gcore user yet?

Learn more about our next-gen CDN

Go to the product page