New feature: raw logs via S3 and FTP
- November 9, 2018
- 2 min read
Raw logs can be helpful in obtaining answers to many questions that arise in the operation of CDN. In particular, they may be used to identify the cause of a problem with content delivery.
Previously, logs were delivered via TCP using Syslog option, and clients had to use special service and software to receive them. It was uncomfortable. Therefore, we have developed more convenient ways of delivering logs.
Where to find the option?
The way you prefer to receive logs is configurable through your control panel—the option can be found in the CDN tab menu.
How to receive logs?
Logs can be uploaded to the client’s storage account via two protocols: S3 and FTP. Appropriate storage for receiving logs is set up and maintained by the clients themselves.
How long do we keep logs?
We keep logs for the past 14 days.
How long logs are kept in the storage?
Since the storage is managed by the clients, they decide by themselves which logs are to be stored and for how long; whether to expand the repository in case of running out of space or just delete the old data.
Receiving logs via S3
You can receive logs via S3 protocol to an Amazon storage or to any other S3 compliant object repository.
Receiving logs via FTP
You can use any standard FTP server to receive information.
How are logs being sent?
Logs from multiple resources can be sent to one or different folders.
Logs are being sent once per hour. If the delivery fails (completely or partially), another attempt will be made an hour later. If logs have not been received for a long time, delivery attempts will be repeated every hour. (But, as noted earlier, we keep logs only for the past 14 days.)
Log delivery statuses
OK. Download successful.
Fail. Logs were not accepted on the client side for 24 hours (all 24 attempts to deliver logs have failed).
Paused. The option is on pause. In this mode settings can be changed. They will be removed only after disabling the option.
What a log looks like?
Log line example:
“89.36.201.215” “-” “-” “[31/Oct/2018:11:30:12 +0000]” “GET /data/cid/storage1/clr/m/dd83f-25ec-4982-8b86-2bd12c2cb/mp4/10/682.m4s HTTP/1.1” “200” “113792” “-” “AppleCoreMedia/1.0.0.16A404 (iPhone; U; CPU OS 12_0_1 like Mac OS X; ru_ru)” “114162” “-” “http” “14.cdn.tv” “0.010” “0.009” “382” “-” “[m9]” “MISS” “113792” “32.122.112.73:80” “2626” “5536” “-” “-” “RO” “Karagandy” “shield_old” “92.223.99.99”
Format of a log file:
log_format gcdn ’”$remote_addr” “-” “$remote_user” “[$time_local]” ’
’”$request” “$status” “$body_bytes_sent” ’
’”$http_referer” “$http_user_agent” ’
’”$bytes_sent” “$sent_http_content_size” “$scheme” “$host” ’
’”$request_time” “$upstream_response_time” “$request_length” “$http_range” ’
’”[m9]” “$upstream_cache_status” “$upstream_response_length” “$upstream_addr” ’
’”$gcdn_api_client_id” “$gcdn_api_resource_id” “$uid_got” “$uid_set” ’
’”$geoip_country_code” “$geoip_city” “$shield_type” “$server_addr”’;
Description of the variables can be found in the Logs section of the knowledge base or in Nginx documentation.
How to activate the option?
Raw logs option is a paid one. To activate it, please contact your account manager or refer to our tech support chat.
Related articles
At Gcore, delivering exceptional streaming experiences to users across our global network is at the heart of what we do. We're excited to share how we're taking our CDN performance monitoring to new heights through our partnership with AVEQ
As streaming demand surges worldwide, providers face mounting pressure to deliver high-quality video without buffering, lag, or quality dips, no matter where the viewer is or what device they're using. That pressure is only growing as audie
Building a startup is tough. Every decision about your infrastructure can make or break your speed to market and burn rate. Your time, team, and budget are stretched thin. That’s why you need a partner that helps you scale without compromis
Subscribe to our newsletter
Get the latest industry trends, exclusive insights, and Gcore updates delivered straight to your inbox.