nxlog on CentOS 6.5: A Review


I want to implement a modern logging architecture using JSON structured data and elasticsearch. Ye Olde Syslog is not good enough. The version of rsyslog that comes as default on CentOS is old and lacks the functionality I need. My three candidates for a new logging clients are: nxlog, syslog-ng, and rsyslog version 8. This is a review of nxlog community edition on CentOS 6.5.

Despite its many strong points, I do not recommend nxlog. I recommend you use one of the other logging clients instead. Perhaps it is a good choice on Windows but not on CentOS. My reasons are below.

In its favor nxlog has excellent primary documentation. This is a sign of quality software. Developers who write good documentation care about the quality of their software. nxlog also has a nice configuration file format.  Here is a snippet to illustrate:

<Input kern>
    Module      im_kernel
    Exec        parse_syslog_bsd();
</Input>

<Output kern_out>
    Module      om_file
    File        "/var/log/kernel.log"
</Output>

<Route kern_route>
    Path        kern => kern_out
</Route>

Like most logging applications it uses the pipe and filter style. nxlog’s format, however, is especially clear. The input, output, and routes are explicit and easy to understand. The “filters” are implicit in the Exec statements which is less desirable, nevertheless, this is the best configuration format I have seen so far. I find it much more clear than even the logstash format.

Good documentation and a good configuration language are so rare that I really wanted to like nxlog. Unfortunately, I simply couldn’t get it to work.  First the rpm install failed because of dependency problems. I had to manually install the three missing packages. The nxlog package didn’t handle that simply task. Once I installed the missing dependencies by hand nxlog installed fine.

Then came the real problem. I opened the default configuration file /etc/nxlog.conf and found that it was completely unusable. Normally, applications provide reasonable defaults so that the software just works. Sometimes you have to edit a line or two in the default configuration if, for example, you have to add a host name. I want software that just works. nxlog doesn’t.

I expected nxlog to function as a drop in replacement for rsyslog. My intent was to customize it from a working base. Instead, I had to figure out how to re-create an entire CentOS logging setup. I didn’t want to give up right away–well, actually I did but decided to persevere anyway–so I read the documentation and added a few rules to handle kernel message and “unix” logs. I then restarted it.

First I had a permission errors because nxlog runs as user nxlog. OK, easy enough to fix. I set all the permissions on the log files and tried again. I waited a bit and … nothing. I tried to send a log using logger but it just hanged. I tried a few more things including sending logs from another host. Still nothing. No logs in the files. I checked the nxlog log itself and saw no errors. When something doesn’t work  and you get no errors you don’t have many options. With regret, I chose the uninstall option.

Perhaps I missed something simple? I did read the install documentation and didn’t see anything.  The unusable default configuration is a show-stopper so it’s time to give rsyslog 8 and syslog-ng a try…



Categories: Software

Tags: , ,

6 replies

  1. Helo Aaron, congrats for your blog, your articles are very good!, last year I was trying to use nxlog, and other options, but finally I used lumberjack(today it’s renamed as logstash-forwarder) https://github.com/elasticsearch/logstash-forwarder

    It could be interesting for your project.

    Regards.

    Like

  2. Hi Aron,

    Thank you for this post. I totally agree with you regarding the weaknesses you broght up. This is one of the reasons nxlog is more popular on Windows. (The other reason is that Linux distros already ship a syslog daemon). Enhancing the linux experience for nxlog users is a task that we will be working on. This will include documentation enhancements, tutorials and more rpm/deb user-friendliness.

    Otherwise I think you gave up a little too early. There are quite a few users who are using in in production across hundreds of linux nodes.
    There is a mailing list and #nxlog on irc.freenode where you could have received some help to get the issues resolved.

    Regards,

    Like

    • Hi Botond,

      Thanks for the comment. As I wrote in my post there is a lot to like in nxlog. I have no doubt that it would have been fast and efficient. My issue was that I was looking for a drop-in replacement for rsyslog and nxlog doesn’t provide that as currently packaged. I don’t know if this is a target use case for you but if it is, only a few additions would enable it fill this role:

      • * Create an rpm package for RHEL 6. Make sure it loads all the dependencies automatically.
      • * Make sure it is SELinux friendly. Many forget this but it’s important.
      • * Include a default config that mimics the standard RHEL syslog settings; i.e., /var/log/messages, /var/log/maillog, etc.

      elasticsearch output would be a nice feature as I’m sure you are already know.

      Cheers,
      Aaron

      Like

  3. Hi Aaron,

    The nxlog deb/rpm packages were designed to be non-intrusive in order to not conflict with the base system. Some people may not prefer to have their already configured local syslogd replaced. Anyone who wishes to do so can do this manually by removing rsyslog and configuring nxlog.conf. Having both nxlog and rsyslog has it’s pros and cons. In the tutorial/documentation we will address both configuration types.
    An rpm/deb package cannot download and install dependencies automatically. This is the task of the package management system. This dependency issue is simply not due the nxlog package and your complaint is not valid IMHO: “I had to manually install the three missing packages. The nxlog package didn’t handle that simply task.” The only thing that can help here is an apt/yum repository which should do what you are after. Unfortunately when installing with the ‘rpm’ command, you will need to resolve all dependencies manually first. The yum/apt repository is on the roadmap.
    IMO the standard rsylog/syslogd settings (i.e. storing logs various logs under /var/log) are just simply an old artifact from 20 years ago. Regardless , we will be providing a default config that will do something like this and would allow the user to easily finetune the configuration if he needs more log files scattered around in /var/log.
    Here is mailing list post that should help: http://www.mail-archive.com/nxlog-ce-users%40lists.sourceforge.net/msg00223.html
    Make sure to have /dev/log a symlink and set the required permissions on the directories.
    You are right about SELinux support.
    Elasticsearch output is on the way.

    Appreciate your comments!
    Regards,
    Botond

    Like

    • Botond,

      Interesting discussion and I am glad to hear your point of view. A few responses…

      • “Having both nxlog and rsyslog has it’s pros and cons”. It could be my own bias but this looks to me like an atypical case. It seems to me that the default should always address the typical case. If a mixed deployment is your target audience’s primary use case then, of course, that makes perfect sense.
      • “your complaint is not valid” Well, inaccurate, yes but not invalid. 🙂 When my users say the “Internet is down” it isn’t wrong even if the problem is actually the router or the DNS server. That is just how it looks to them. What I wanted to convey was that installing the nxlog package did not automatically install the dependencies as it does when I install from the default repository. Still, you are technically correct and my wording was sloppy. I should have said that nxlog should be included in EPEL (if that’s possible) or at least in a custom repository so that the user experience is what I was expecting.

      • “an old artifact from 20 years ago”. I certainly agree with this in principle. Still, the /var/log structure is part of the Linux FHS and LSB standard (http://www.pathname.com/fhs/pub/fhs-2.3.html#VARLOGLOGFILESANDDIRECTORIES). I am curious what you would recommend in its place? In my logging architecture I plan on leaving that old style in place while adding a modern, structured-data, client/server approach on top of it.

      As for SELinux, I suspect that it might have been the cause of some of the problems I had with nxlog. Recently, I could not get the rsyslog elasticsearch plugin to work after trying everything. By chance I checked my audit logs and there it was. SELinux errors can be tricky to identify.

      Thanks,
      Aaron

      Like

  4. Hi Aaron,

    I’m not saying storing logs in /var/log is bad. The thing that is bad with this is:

    In many cases you get mail.err/mail.info/mail.log/mail.warn and various other files scattered around instead of having just 2-3 log files.
    The format should be also revised so that the logfile has an ISO timestamp with the year and also the severity/facility (not numeric ) but MAIL.INFO or something similar. I can grep if I want.

    People are afraid to make changes that can break things, things that work, though are awkward to use in the 21st century. That’s why the rsyslog conf has that awkward syntax because they wanted to retain compatibility with the original syslogd config file format.

    One of the main purposes of nxlog is centralized log collection. This is usually mandated by the security team in an organization. Thus when the sysadmin is given an nxlog installer, she will be less resistant to solutions that do not interfere with the base system, i.e. no need to replace rsyslog, just configure nxlog to collect files written into /var/log and ship that to the central log server.

    Regards,
    Botond

    Like

Share Your Ideas

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: