Install from Apt Repository (For Bionic)
curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-bionic-td-agent2.5.sh | sh
Launch Daemon
sudo systemctl start td-agent.service
Restart Daemon
sudo systemctl restart td-agent.service
Link: https://docs.fluentd.org/v/0.12/articles/before-install
How to manage Fluentd plugins.
Install fluent-plugin-remote_syslog
td-agent-gem install fluent-plugin-remote_syslog
or
/opt/td-agent/bin/fluent-gem install fluent-plugin-remote_syslog
Link: https://docs.fluentd.org/deployment/plugin-management
Main configuration file:
nano /etc/td-agent/td-agent.conf
Logfile fluentd:
less /var/log/td-agent/td-agent.log
The configuration file consists of the following directives:
source directives determine the input sources.
match directives determine the output destinations.
filter directives determine the event processing pipelines.
system directives set system wide configuration.
label directives group the output and filter for internal routing
include directives include other files.
Link: https://docs.fluentd.org/v/0.12/configuration/config-file
Official documentation: https://docs.fluentd.org/v/0.12/
Where all the data come from.
# http turns fluentd into an HTTP endpoint to accept incoming HTTP messages
<source>
@type http
port 9880
</source>
# forward turns fluentd into a TCP endpoint to accept TCP packets
<source>
@type forward
port 24224
</source>
# read Nginx access logs using in_tail
<source>
@type tail
path /path/to/input/file
format nginx
keep_time_key true
</source>
# read myapp.log access logs using in_tail in apache2 format
<source>
@type tail
path /var/log/myapp.log
pos_file /var/log/td-agent/myapp.pos.log
tag myapp
format apache2
</source>
Tell fluentd what to do.
# Match events tagged with "myapp.access" and
# store them to /var/log/fluent/access.%Y-%m-%d
<match myapp.access>
@type file
path /var/log/fluent/access
</match>
# Match events tagged with "myapp" and
# send them to remote server syslog with tag fluentd
# rsyslog must be configured for accept it
<match myapp>
@type remote_syslog
host 123.124.125.126
port 514
protocol udp
program fluentd
tag fluentd
</match>
* remote_syslog - plugin for output to remote syslog serivce.
Link: https://github.com/dlackty/fluent-plugin-remote_syslog
The directives in separate configuration files can be imported. The @include directive supports regular file path, glob pattern, and http URL conventions. Also the directive can be used under sections to share the same parameters.
# absolute path
@include /path/to/config.conf
# if using a relative path, the directive will use
# the dirname of this config file to expand the path
@include extra.conf
# glob match pattern
@include config.d/*.conf
# http
@include http://example.com/fluent.conf
flog (file logger) is a program that reads input from STDIN and writes to a file.
Install
sudo apt install flog
Or the newest version
wget https://github.com/mingrammer/flog/releases/download/v0.4.3/flog_0.4.3_linux_amd64.tar.gz
tar xvzf flog_0.4.3_linux_amd64.tar.gz
Usage:
pipeline | flog [-t] logfile
# Send apache logs to /var/log/myapp.log
./flog -t log -f apache_combined -o /var/log/myapp.log -n 200
-t, --type string log output type
-f, --format string log format
-o, --output string output filename
-n, --number integer number of lines to generate.