Logs uploader (near real-time)

Logs Uploader is a feature that enables an automatic export of CDN resource logs to your storage in near real-time.

Exported logs contain information about user requests sent to cache servers and pre-cache servers (if you have the Origin shielding feature enabled).

Logs uploader settings

In this section, you can find general information about log settings, statuses, and how to configure logs exporting for different storage types.

Log format example

      Copy
      
    "$remote_addr" "-" "$remote_user" "[$time_local]" "$request" "$status"  
"$body_bytes_sent" "$http_referer" "$http_user_agent" "$bytes_sent"  
"$edgename" "$scheme" "$host" "$request_time"  
"$upstream_response_time" "$request_length" "$http_range" "[$responding_node]"  
"$upstream_cache_status" "$upstream_response_length" "$upstream_addr"  
"$gcdn_api_client_id" "$gcdn_api_resource_id" "$uid_got" "$uid_set"  
"$geoip_country_code" "$geoip_city" "$shield_type" "$server_addr" "$server_port"  
"$upstream_status" "-" "$upstream_connect_time" "$upstream_header_time"  
"$shard_addr" "$geoip2_data_asnumber" "$connection" "$connection_requests"  
"$http_traceparent" "$http_x_forwarded_proto" "$gcdn_internal_status_code" "$ssl_cipher"  
"$ssl_session_id" "$ssl_session_reused" "$sent_http_content_type" "$tcpinfo_rtt" 
"$server_country_code" "$gcdn_tcpinfo_snd_cwnd" "$gcdn_tcpinfo_total_retrans" "$gcdn_rule_id" 

It’s OK if you find a field that’s not listed in the example. We occasionally add new fields to the end of the line. If some fields are added to logs, you’ll receive an email about the update.

Log fields

The following table contains a complete list of available log fields. Fields formatted in italics relate to our internal CDN system, so you can ignore them.

You can check other fields—they can be helpful for traffic analysis or statistics.

Configure logs for export

You can enable and set up the Logs uploader feature in the Gcore Customer Portal on the Gcore CDN page.

To access log settings, navigate to Logs > Logs uploader and configure the feature as described in the following steps.

Step 1 (optional). Include empty logs

Keep the Do not send empty logs option selected if you don't want to receive empty logs. Otherwise, uncheck it.

Step 2 (optional). Enable origin shielding

If you are using the origin shielding feature, you’ll see the Add logs from origin shielding checkbox when configuring Logs uploader.

We recommend that you select this option as it ensures that the logs report will include both requests to cache services and requests to the pre-cache server. Thus, you’ll receive more detailed information on resource usage. To include these logs, select the checkbox to enable Include logs from origin shielding.

Step 3. Select log fields

Choose which log fields you want to include in the exported report. By default, all fields are selected.

Step 4. Configure a storage provider

Amazon

Follow these instructions to export logs to AWS storage:

1. In the Storage provider, select Amazon.

2. Provide your access key ID and secret access key, which together form long-term AWS credentials.

3. (Optional). Choose a storage region. While the region is often determined automatically, we recommend specifying it to ensure that your logs are exported successfully.

4. Specify the name of a bucket where you want to export CDN logs.

5. (Optional). Enter a folder name if you want to export logs to a specific folder within a bucket.

6. Click Save changes to apply the updates.

OSS

Follow these instructions to export logs to configure logs for Alibaba Cloud Object Storage Service (OSS):

1. In the Storage provider, select OSS.

2. Provide your access key ID and secret access key. Check the official OSS documentation for details.

3. (Optional). Choose a storage region. While the region is often determined automatically, we recommend specifying it to ensure that your logs are exported successfully.

4. Specify the name of a bucket where you want to export CDN logs.

5. (Optional). Enter a folder name if you want to export logs to a specific folder within a bucket.

6. Click Save changes to apply the updates.

Gcore\Other

Follow these instructions to export logs to Gcore Object Storage or any S3 storage of your choice:

1.In the Storage provider, select Gcore or Other.

2. Specify a hostname—a name that’s assigned to a storage server within a network and is used instead of an IP address. In you’re using Gcore Storage, you can find your hostname in the storage details section.

3. Provide your access key ID and secret access key. You can find this information in the Details of the storage.

4. Specify the name of a bucket where you want to export CDN logs.

5. (Optional). Enter a folder name if you want to export logs to a specific folder within a bucket.

6. Click Save changes to apply the updates.

How near real-time log exporting works

Logs are achived and exported to your endpoint every five minutes. If CDN servers are not requested and you didn’t select the Do not send empty logs checkbox when configuring Logs uploader, an empty log file (± 20 bytes) will be sent to your storage.