Install and configure Fluent Bit
Fluent Bit is a log shipping tool. This means that it is a service that collects logs from a device and sends them to an external storage.
In Fluent Bit’s Settings, you specify a source of the logs you need, for example, logs of a specific service, internal storage, TCP port, or OS events. You must indicate a destination for log export—Gcore’s Logging servers. After that, Fluent Bit works automatically: Once a required log is recorded in the system, the log shipper transfers it to our storage, and the log appears on OpenSearch Dashboards.
Install
Fluent Bit’s official documentation provides up-to-date installation guides for different operating systems. Open this link and select your OS. Simply follow the steps in the relevant installation manual.
Configure
The Fluent Bit configuration file consists of two sections: INPUT and OUTPUT. INPUT determines the source of the logs you need to collect, and OUTPUT indicates the destination where they will be sent. Here’s how to set up the configuration file correctly:
- Open the
fluent-bit.conffile and add the following data:
Customize the highlighted values:
- tail: Data source
- /var/log/syslog: File path
- laas-example.gcore.com:443: Kafka Endpoint on the Logging page
- yourlogin.yourtopic: Your username on the Logging page and your topic name separated with a dot (.)
- yourlogin: Your username on the Logging page
- yourpassword: Your password
For more information on how to add INPUT for different log sources, go to the "Input" section of the Fluent Bit documentation and click the log source you need.
For example, if you want to gather logs from a specific file, open the guide for Tail. Tail is a utility on UNIX-like systems used to display the tail end of a file. It helps Fluent Bit to read changes in the log file. You need to fill in the INPUT so that it corresponds with Tail.
- [OUTPUT]: Type of process (export).
- name: Servers where logs will be delivered (Kafka servers).
- brokers: Server(s) where logs will be exported to.
- topics: Topic(s) where logs will be exported to.
- rdkafka.security.protocol: Security protocol that encrypts data to protect it from theft.
- rdkafka.sasl.mechanism: Authentication mechanism that helps to verify a login and a password entered to sign into your logs storage.
- rdkafka.sasl.username: The username that helps to verify the sender.
- rdkafka.sasl.password: The password that helps to verify the sender.
Save the changes in the
fluent-bit.conffile.Restart Fluent Bit, and it will begin sending logs to your Gcore Logging storage.
No timestamps
Some services (for example, nginx) transmit timestamps of logs in a non-standard format. In this case, OpenSearch Dashboards will show logs without their date and time. If this happens, copy the string below to the OUTPUT section. This will ensure that Fluent Bit forwards date and time in a different format that is suitable for services like nginx.
timestamp_format iso8601Was this article helpful?
Not a Gcore user yet?
Discover our offerings, including virtual instances starting from 3.7 euro/mo, bare metal servers, AI Infrastructure, load balancers, Managed Kubernetes, Function as a Service, and Centralized Logging solutions.