How to send JSON log messages to RabbitMQ from syslog-ng


In this post I will show you how to configure syslog-ng to send logs in JSON format in the body of an AMQP message. I use RabbitMQ as my message broker but this method should work with any AMQP broker supported by syslog-ng.

I show a basic, working configuration. It should be enough to get you started and that you can to adapt to your needs.

Skill Level: Intermediate

Prerequisites

  • You have RabbitMQ installed and running, and you understand the AMQP model.
  • You have syslog-ng installed and running. You understand how to configure syslog-ng sources and destinations.
  • syslog-ng is running on Linux. You understand Linux administration commands.

Software Versions

  • RabbitMQ 3.6.6. Earlier versions will almost certainly work as well. Later versions will likely keep working until there is some major, breaking change.
  • syslog-ng 3.8.1. Earlier but still recent versions should work (such as 3.4.x, and 3.5.x).
  • AMQP 0.9.1. The syslog-ng documentation states that it supports AMQP 1.0 but I did not test it.

Steps

  1. Create a direct, durable exchange in RabbitMQ (using its web UI ) called “test”.
  2. Stop the syslog-ng service (for example, “systemctl stop syslog-ng”)
  3. Add the syslog-ng destination definition shown below to /etc/syslog-ng/syslog-ng.conf. If you use configuration files under /etc/syslog-ng/conf.d then put the destination definition in the appropriate configuration file.
  4. In the same syslog-ng configuration file, add a log definition to connect your sources to the new AMQP destination. Mine looks like this “log { source(s_sys); destination(d_rabbit); };” but yours will certainly differ.
  5. Check the syntax of your configuration file using the command “syslog-ng –syntax-only”. This command should produce no output when your configuration has the correct syntax.
  6. Create a RabbitMQ message receiver test client in the language of your choice. The RabbitMQ web site has many examples. You can adapt the “Hello World” example for this test. Make sure that the client prints the message body it receives to the console.
  7. Choose an AMQP routing key for this test. For example, “system.syslog-ng”. Configure your test client and syslog-ng to use this key. Your client should bind a queue to the “test” exchange with this key.
  8. Start your test client. It should go into listen mode and be ready to receive messages.
  9. Run syslog-ng in foreground mode using the command, “syslog-ng -Fevd”. If your configuration is correct and syslog-ng has the correct support for JSON and AMQP then you should not see any AMQP-related error messages.
  10. To complete this test your system must send a log message. Depending on your system this may be immediate or may take a while.
    • You can use the Linux logger program to manually send a log message. This is, however, problematic because syslog-ng is running in the foreground. You will need to open another shell via “screens” or through another remote session.
  11. When a log message is sent you should see it in the syslog-ng console output (with no errors), and you should see the JSON in your test client.
  12. Once this basic example works, you can adapt it to your environment–change the exchange and routing key, change the JSON structure, etc. And then you can start syslog-ng as a service.

AMQP Destination

Below is the destination definition that worked for me:

destination d_rabbit {
    amqp(
    vhost("/")
    host("[hostname or IP address of your RabbitMQ server]")
    port(5672)
    exchange("test")
    exchange-type("direct")
    routing-key("system.syslog-ng")
    body("$(format-json date=datetime($ISODATE) pid=int64($PID) program=$PROGRAM message=$MESSAGE facility=$FACILITY host=$FULLHOST priorityNum=int64($LEVEL_NUM) priority=$LEVEL)")
    username("guest")
    password("guest")
    persistent(no)
    );
};

The destination name “d_rabbit” is an example. You can use whatever name you want except for “d_amqp” which comes from the syslog-ng documentation and caused syslog-ng to fail.

vhost, host, and port come from your RabbitMQ server configuration. At a minimum you need to include your RabbitMQ server hostname or IP address. exchange, exchange-type, routing-key, and persistent are standard AMQP options.  The values I’ve included here are simple ones for this example. You can use whatever values make sense for your architecture (see the AMQP configuration options link below for more options).

You must include username and password even if your RabbitMQ is not configured for different users. If your RabbitMQ server is configured for user security then put in the appropriate information here. If it is not then you can leave “guest” as the values. The destination will fail without these.

They key option is body. I wanted the log in JSON format and in the body of the AMQP message instead of having key-value pairs in the AMQP header. I also customized the JSON to include the property names and log values that I was interested in. After format-json the X=Y statements tell syslog-ng what the JSON property names should be (the ‘X’) and what its value should be (the ‘Y’).

For example, date=datetime($ISODATE) means include a JSON property called date and set its value to the current time in ISO 8601 date format. All JSON properties in syslog-ng will be of type string by default. The function int64() makes the value an integer instead. The other type options are listed here.

You can include whatever values you want and give them whatever JSON property names you want by changing the X=Y statement. See the macros link below for the log data values available in syslog-ng.

My configuration produces JSON that looks like this:


{
    "program": "xe-daemon",
    "priorityNum": 5,
    "priority": "notice",
    "pid": 1810,
    "message": "Running CollectDisk ...",
    "host": "myhost",
    "facility": "user",
    "date": "2017-02-03T15:21:48-05:00"
}

Useful Links



Categories: Software

Tags: ,

Share Your Ideas

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: