Logging to the syslog from a java application

Every application needs logging, it can help you during development and when debugging those annoying things that do not work in production. One question is where to put the logging events. All linux servers use a system log to log events that take place on the operating system level. You can find logs for the kernel, deamons, user actions and a lot of other items. The nice part about system logging is that maintenance people will always know where to look and that it is possible to use one server for logging.

At the moment I am on a project that uses a fair amount of servers. We have more than 20 servers for the different environments and a lot of components to investigate when trying to find problems. Think about squid logs, apache httpd logs, tomcat logs and more. To make this doable, we have a syslog server.

An application running in Tomcat does not log to the system log by default. In our situation, we want our components to log to different files. Syslog has a special facility that makes it easy to do just that.

This blog post discusses the different parts of configuring system logging from a java application using the well known log4j.

syslog-ng introduction

This very short introduction is based on this article.

With syslog you can log to the local server but also to a remote server. It is also possible to do them both using a little bit of configuration. The following image gives an idea about how a logging environment can be used. (took it from the mentioned article)

Syslog consists of a few elements.

• Sources – Where can logs come from, the facilities.
• Filters – Used to for instance log events based on their severity. I used this to filter on Facility
• Destinations – Where to actually send the logs to, a file a remote host.
• Logs – Combine the sources, filters and destinations

Before we can talk about the configuration I need to tell you something about Facilities. A facility defines the source of the log. You can think about kernel messages, user-level messages, mail system and many more. The javadoc of the appender shows the possible values for these Facilities. There are also a few special facilities, these are defined as local0, local1, etc. These are the ones you can use for your own application. We use one of these for every different logfile we want.

Configure syslog-ng

The mentioned components are easily identified in the configuration of the syslog. The first thing to configure is udp access to the syslog. To enable udp logging, open the file “/etc/syslog-ng/syslog-ng.conf” and make sure to configure the sources part like the following block:

<br>
######<br>
# sources</p>
<p># all known message sources<br>
source s_all {<br>
# message generated by Syslog-NG<br>
internal();<br>
# standard Linux log source (this is the default place for the syslog()<br>
# function to send logs to)<br>
unix-stream("/dev/log");<br>
# messages from the kernel<br>
file("/proc/kmsg" log_prefix("kernel: "));<br>
# use the following line if you want to receive remote UDP logging messages<br>
# (this is equivalent to the "-r" syslogd flag)<br>
udp();<br>
};<br>

Next up is configure the log files to which the syslog events will be send. First we configure the destination:

<br>
# destinations<br>
destination df_local0 { file ("/var/log/cms.log"); };<br>
destination df_local1 { file ("/var/log/site.log"); };<br>
destination df_local2 { file ("/var/log/importer.log"); };<br>

This code block configures three destinations with the names df_local0/1/2 and points them to the mentioned log files. Now we have to filter the incoming log events. We use the facilities that we discussed to filter the incoming events. There are lots of other options, but this is very easy to use. The following lines show the configuration of the filters.

<br>
# filters<br>
filter f_local0 { facility(local0); };<br>
filter f_local1 { facility(local1); };<br>
filter f_local2 { facility(local2); };<br>

Finally we configure the logs, these are combinations of a source, a destination and a filter.

<br>
log {<br>
source(s_all);<br>
filter(f_local0);<br>
destination(df_local0);<br>
};<br>
log {<br>
source(s_all);<br>
filter(f_local1);<br>
destination(df_local1);<br>
};<br>
log {<br>
source(s_all);<br>
filter(f_local2);<br>
destination(df_local2);<br>
};<br>

Do not forget to restart the syslog service after configuration changes

Testing to see that it works

Before you start messing around with log4j, you can use a few tools to send log events to the syslog server. There are command line utilities available on all linux systems. Check this post for more information. The following command sends a message:

nc 192.168.1.1 514 <<< "<14>User Info msg from remote through TCP."

Be sure to check the port of the server (514 is the default).

Configure log4j

The final step is to send syslog events from your java application. Log4j comes out of the box with a SyslogAppender. There is one requirement to be able to use log4j with sys logging. You need to have udp enabled like we discussed in the configuration section. Most of this comes from the mentioned blog post.

An example of log configuration for log4j :

<br>
log4j.rootLogger=INFO, SYSLOG</p>
<p>log4j.appender.SYSLOG=org.apache.log4j.net.SyslogAppender<br>
log4j.appender.SYSLOG.syslogHost=127.0.0.1<br>
log4j.appender.SYSLOG.layout=org.apache.log4j.PatternLayout<br>
log4j.appender.SYSLOG.layout.conversionPattern=%d{ISO8601} %-5p [%t] %c{2} %x - %m%n<br>
log4j.appender.SYSLOG.Facility=LOCAL1<br>

That is it, now you have a java application logging to the syslog. Feedback about improvements is welcome.