Centralizing syslog data

Overview

I recently set up a few servers out in “the cloud.” I needed a way to ship all the logs generated by these hosts to my splunk instance. Since shipping raw syslog data across the internet is a bad idea, this also needed to be secure. I opted for an ssh tunnel from my splunk server over to each of servers on the internet. One could also configure OpenVPN, however I wanted to try a different route.

Setup

Lets start with the apache web server instance. I wanted a copy of web access logs to be searchable and indexed. None of the documentation I read shows how to get access logs over to a syslog facility. I solved this using a pipe and the logger command.

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" accesslogs
CustomLog "|/usr/bin/tee -a /var/log/apache2/access.log |/usr/bin/logger -t httpd -i -p local4.info" accesslogs

Add the two lines above to your apache2.conf file. The first line sets up a custom format for the logs called accesslogs. The second line uses the CustomLog directive to both save the logs locally on the server in /var/log/apache2/access.log as well as send a duplicate to the local4 facility of syslog. You can read about the flags used the LogFormat directive here and the logger man page here.

The second piece to the logging puzzle is the syslog, or rsyslog configuration in my case. My Debian install shipped with rsyslog installed by default. I added the following line to the bottom of /etc/rsyslog.conf:

*.* @@127.0.0.1:50514

Basically, this tells rsyslog to send all messages to the tcp destination of 127.0.0.1 on port 50514. Note the two “@” signs before the IP. This is what tells rsyslog to use tcp. This is important because we will be configuring the ssh tunnel next for the logs to be sent over.

While still on the remote host, add a new user account. This user account will be used for the splunk server to ssh to and establish a secure tunnel. You will also want to configure keys and delete the users password after account creation. This will allow the connection to be re-established without intervention and minimize the risk for an attack through this account.

On the splunk server create a new file called rsyslog-ssh-tunnel.conf or similar in /etc/init/. My splunk server is running on Debian. You will also need the autossh package installed which can be found in aptitude. Contents of my configuration file are below:

# This service maintains a remote ssh tunnel for rsyslog

start on net-device-up IFACE=eth0
stop on runlevel [01S6]

	respawn
respawn limit 5 60

script
su autossh -c 'exec autossh -nNT -R 50514:localhost:1514 <username>@<ip>'
end script

I created an account on the splunk server called autossh which I configured with ssh keys for my web server. This short script brings up and monitors the ssh tunnel as the autossh user. Also note the first port in the connection string, 50514. This is what is configured remotely on the web server as the syslog destination. The second port, 1514 is the local port. This is where the syslog traffic will source from on the splunk server. Simply add 127.0.0.1 as a log source on port 1514 to have logs from the web server in the splunk instance.

Testing

Once all the configuration is in place, it’s a good idea to test everything first. You will need to accept the public ssh key on the first connection attempt if you did not manually copy them over. Also bring up the tunnel as the autossh user first, then restart rsyslog on the remote side, and finally restart apache. Check for logs by running a packet capture on the loopback interface on port 1514 on the splunk server. Once this has been completed, apache access logs will now be searchable; along with other general system logs.