New feature: raw logs via S3 and FTP

New feature: raw logs via S3 and FTP



Raw logs can be helpful in obtaining answers to many questions that arise in the operation of CDN. In particular, they may be used to identify the cause of a problem with content delivery.

New feature: raw logs via S3 and FTP

Previously, logs were delivered via TCP using Syslog option, and clients had to use special service and software to receive them. It was uncomfortable. Therefore, we have developed more convenient ways of delivering logs.

Where to find the option?

The way you prefer to receive logs is configurable through your control panel—the option can be found in the CDN tab menu.

New feature: raw logs via S3 and FTP

How to receive logs?

Logs can be uploaded to the client’s storage account via two protocols: S3 and FTP. Appropriate storage for receiving logs is set up and maintained by the clients themselves.

How long do we keep logs?

We keep logs for the past 14 days.

How long logs are kept in the storage?

Since the storage is managed by the clients, they decide by themselves which logs are to be stored and for how long; whether to expand the repository in case of running out of space or just delete the old data.

Receiving logs via S3

You can receive logs via S3 protocol to an Amazon storage or to any other S3 compliant object repository.

New feature: raw logs via S3 and FTP

Receiving logs via FTP

You can use any standard FTP server to receive information.

New feature: raw logs via S3 and FTP

How are logs being sent?

Logs from multiple resources can be sent to one or different folders.

Logs are being sent once per hour. If the delivery fails (completely or partially), another attempt will be made an hour later. If logs have not been received for a long time, delivery attempts will be repeated every hour. (But, as noted earlier, we keep logs only for the past 14 days.)

Log delivery statuses

OK. Download successful.

Fail. Logs were not accepted on the client side for 24 hours (all 24 attempts to deliver logs have failed).

Paused. The option is on pause. In this mode settings can be changed. They will be removed only after disabling the option.

What a log looks like?

Log line example:

“89.36.201.215” “-” “-” “[31/Oct/2018:11:30:12 +0000]” “GET /data/cid/storage1/clr/m/dd83f-25ec-4982-8b86-2bd12c2cb/mp4/10/682.m4s HTTP/1.1” “200” “113792” “-” “AppleCoreMedia/1.0.0.16A404 (iPhone; U; CPU OS 12_0_1 like Mac OS X; ru_ru)” “114162” “-” “http” “14.cdn.tv” “0.010” “0.009” “382” “-” “[m9]” “MISS” “113792” “32.122.112.73:80” “2626” “5536” “-” “-” “RO” “Karagandy” “shield_old” “92.223.99.99”

Format of a log file:

log_format gcdn  ’”$remote_addr” “-” “$remote_user” “[$time_local]” ’

       ’”$request” “$status” “$body_bytes_sent” ’

       ’”$http_referer” “$http_user_agent” ’

       ’”$bytes_sent” “$sent_http_content_size” “$scheme” “$host” ’

       ’”$request_time” “$upstream_response_time” “$request_length” “$http_range” ’

       ’”[m9]” “$upstream_cache_status” “$upstream_response_length” “$upstream_addr” ’

       ’”$gcdn_api_client_id” “$gcdn_api_resource_id” “$uid_got” “$uid_set” ’

       ’”$geoip_country_code” “$geoip_city” “$shield_type” “$server_addr”’;

Description of the variables can be found in the Logs section of the knowledge base or in Nginx documentation.

New feature: raw logs via S3 and FTP

How to activate the option?

Raw logs option is a paid one. To activate it, please contact your account manager or refer to our tech support chat.

New feature: raw logs via S3 and FTP

Subscribe
to our newsletter

Get the latest industry trends, exclusive insights, and Gcore
updates delivered straight to your inbox.