After a lot of experimentation I’ve decided on logging architecture that looks like this:
This diagram shows the logical design. I will explain the components.
These are switches, routers, appliances, and so on that can only send syslog format messages to a server. These go to the log rules engine (OSSEC) for processing.
Hosts are servers that can run the logging rules engine client (in my case OSSEC) and which can send syslog format messages to a central log server. This double path is needed because OSSEC does not provide an alternate means of sending logs to the rules engine. I use Syslog-NG as my logging client instead of rsyslog.
Log Rules Engine
This component is an automated log reader. It reads the logs and if it sees anything notable it generates an alert.This is a necessity given the enormous log volume even a small IT infrastructure can generate.
I use OSSEC and Syslog-NG for this role. Syslog-NG acts as the syslog server for the devices and it also forwards the device logs to the raw log storage server. Syslog-NG stores the device logs in files which OSSEC can process.
OSSEC provides the log rules engine to generate the alerts.
Raw Log Storage
This component receives the raw logs from the hosts and devices as well as the alerts. It archives them for compliance and for other eventualities such as legal evidence. I use Syslog-NG as the log server. In addition to archiving the raw logs, Syslog-NG also forwards them to the translator.
This component parses the raw logs and converts them into structured JSON. Different logs have different JSON structures. For example, iptables and postfix each have a unique format that must be parsed so the JSON fields can be correctly populated. This requires custom parsers for each log type.
I haven’t found an ideal solution for this component yet. logstash is problematic for reasons I wrote about earlier. I tried a recent version of rsyslog but its documentation was impenetrable (even by open source standards). fluentd was recommend so I will give it a try. I may use Syslog-NG as the parser and then a custom component to connect it Elasticsearch since Syslog-NG doesn’t support Elasticsearch natively.
Structured Log Storage
This component holds the structured logs. Structuring the logs makes them much easier to monitor, search, and process. I use Elasticsearch for this component with Kibana as a GUI.