Log management is the collection of self-generated data from IT hardware devices and software applications. The collection of this data can contain useful information about business processes such as the number of errors on a website or even a security issue that displays the number of failed attempts to access a perimeter router.
Many organizations conduct log management practices to meet regulatory compliance in their respected industries. For instance, the Graham-Leach-Bliley (GLB) Act provides compliance guidelines to those organizations in the financial industry, while the Healthcare Insurance Portability and Accountability Act (HIPAA) has a direct effect on the health care industry. In addition, the Sarbanes-Oxley (SOX) Act offers protection guidelines to publicly traded companies’ financial systems and the Federal Information Security Management Act (FISMA) aids in the protection of the federal government’s information systems.
However, the use of log management practices serves more than just a vehicle to satisfy compliance requirements, it can also be used to assist in the fault management process to aid in troubleshooting at the tier 1 level. In addition to fault management, performance analysis is another useful area for log data to obtain information about server CPU, memory, disk and even network throughput. Lastly, log data can conduct trend analysis to view peak usage stats such as the “top”10s as well as aid in the detection of security incidents and violations.
When implementing a log management solution, there are various types of solutions, architectures and definitions. Here are a few you should be familiar with:
- Sinkhole – Traditional single “syslog” server that “receives” remote logs from one or more sources
- Hierarchy – Multi-Tiered sinkhole’s divided by department, network (vlan), or other logical fashion such as accounting, marketing and engineering to collect log data.
- Aggregator – Usually located at the top of a hierarchy, where major functionality such as alerting, reporting, searching and correlations occur.
- Distributed – Independent log repositories, may be searchable/accessible from a central location.
- Store and Forward – logs are written to a local disk or network disk to be spooled and sent later.
- Streaming – Real-time distribution of log data to a remote logging server as they are being generated.
- Agent Based – Operating systems that do not support remote logging often require assistance from software to send log data. Even those operating systems that are capable of sending log data may use agents to send out specific data in a secure manner to a logging server.
- Agent Less – Systems do not send log data directly to the log server, the logger itself obtains the data via secure file copy (store & grab) or WMI (Windows Management Interface)
- Combo – Most mature log management infrastructures use part or all the above in some way or fashion.
There are a number of useful tools in both the open-source and commercial space that can assist in the creation of a log management solution or the upgrade of an existing solution. In the open-source area, the following solutions are:
- Syslog-ng – Unix based tool. Swiss army knife of log management. Can read any file and “tail” it to the network. Commercial versions available (sinkhole/ forwarding agent)
- Rsyslog – Like syslog-ng , with enhanced filtering, encryption, buffering
- OSSEC – Host Based or Server Based SIM/IDS (Aggregator/Agent)
- –SEC.pl – Simple Event Correlator (Aggregator)
- –PHP-Syslog, MySQL – PHP interface to logs in a database
- –Lasso – Agent-less collection agent for Windows (WMI based)
While in the commercial realm, there are many formidable solutions available, but are just a here are a few notable ones:
Splunk – In the “Pro” column, Splunk provides relevant search information very fast due to its use of data indexing. It has a wide support for various operating systems such as Windows, Mac OS X and Linux-based systems and it is extremely easy to use. One can also have used the software at no cost up to 500MB of log data. In the “Con” column, Splunk seem to have quick development cycles that requires numerous software updates and the advanced features, such as “app” development has a bit of a steep learning curve.
Log Logic – Log Logic is an appliance-based solution that is also fast and has a wide operating system support based. As for the “Cons”, the cost is a bit high due to its appliance only option and it lacks user specific customization.
LogRhythm – LogRhythm has the ability to collect any type of log data regardless of source and the ability to collect log data with or without installing an agent on the log source device. As for the “Cons”, its use of a database backend may cause insertion delay if the events per second are too high for the setup. This may lead to a delay to access information.
Conclusion
The difference between a log management solution and other types of monitoring tools is that the data is already available on your devices and applications; it is just a matter of setting it up, collecting and using it. In addition, log management is considered an industry and security best practice regardless if your organization has to meet regulatory compliance or not.
One should budget accordingly for the tools selected, even free tools have a cost when factoring hardware and storage components. Some vendors of commercially available tools publish their cost, while others may not, but there is a value on obtaining their professional services for initial deployment at an added cost. It allows for better long-term planning and it is helpful for the initial setup and deployment phases.
