An Open Source Centralized Logging Architecture


After a lot of experimentation I’ve decided on logging architecture that looks like this:

Log_Architecture

This diagram shows the logical design. I will explain the components.

Devices

These are switches, routers, appliances, and so on that can only send syslog format messages to a server. These go to the log rules engine (OSSEC) for processing.

Hosts

Hosts are servers that can run the logging rules engine client (in my case OSSEC) and which can send syslog format messages to a central log server. This double path is needed because OSSEC does not provide an alternate means of sending logs to the rules engine. I use Syslog-NG as my logging client instead of rsyslog.

Log Rules Engine

This component is an automated log reader. It reads the logs and if it sees anything notable it generates an alert.This is a necessity given the enormous log volume even a small IT infrastructure can generate.

I use OSSEC and Syslog-NG for this role. Syslog-NG acts as the syslog server for the devices and it also forwards the device logs to the raw log storage server. Syslog-NG stores the device logs in files which OSSEC can process.

OSSEC provides the log rules engine to generate the alerts.

Raw Log Storage

This component receives the raw logs from the hosts and devices as well as the alerts. It archives them for compliance and for other eventualities such as legal evidence. I use Syslog-NG as the log server. In addition to archiving the raw logs, Syslog-NG also forwards them to the translator.

Log Translator

This component parses the raw logs and converts them into structured JSON. Different logs have different JSON structures. For example, iptables and postfix each have a unique format that must be parsed so the JSON fields can be correctly populated. This requires custom parsers for each log type.

I haven’t found an ideal solution for this component yet. logstash is problematic for reasons I wrote about earlier. I tried a recent version of rsyslog but its documentation was impenetrable (even by open source standards). fluentd was recommend so I will give it a try. I may use Syslog-NG as the parser and then a custom component to connect it Elasticsearch since Syslog-NG doesn’t support Elasticsearch natively.

Structured Log Storage

This component holds the structured logs. Structuring the logs makes them much easier to monitor, search, and process. I use Elasticsearch for this component with Kibana as a GUI.



Categories: DevOps

Tags:

2 replies

  1. Aaron – thanks for providing an excellent resource for folks interested in security, logging and monitoring. Do check out another blog http://www.openopsiq.com. Also have you looked at Apache Flume in place of Logstash? Large organizations like Netflix have published some very interesting architectures related to real-time monitoring of large platforms. Check out Netflix Suro.

    http://openopsiq.com/2013/12/15/announcing-suro-backbone-of-netflixs-data-pipeline/

    Also you may like the report by a Nato Cybersecurity Center of Excellence on various open source tool options.

    http://openopsiq.com/2013/12/30/comparative-analysis-of-open-source-log-management-solutions-for-security-monitoring-and-network-forensics/

    Like

Share Your Ideas

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: