Log monitoring/analysis

Log Analysis (or log inspection) is done inside OSSEC by the logcollector and analysisd processes. The first one collects the events and the second one analyzes (decodes, filters and classifies) them.

It is done in real time, so as soon as an event is written OSSEC will process them. OSSEC can read events from internal log files, from the Windows event log and also receive them directly via remote syslog.

What is log analysis?

Inside OSSEC we call log analysis a LIDS, or log-based intrusion detection. The goal is to detect attacks, misuse or system errors using the logs.

LIDS - Log-based intrusion detection or security log analysis are the processes or techniques used to detect attacks on a specific network, system or application using logs as the primary source of information. It is also very useful to detect software misuse, policy violations and other forms of inappropriate activities.

Quick Facts

  • How often are logs monitored?

    • In real time.

  • Where are the events analyzed?

    • In the manager.

  • How long are they stored?

    • For as long as your policy dictates (it is user configurable).

  • Where does this help me with compliance?

    • (PCI DSS, etc) It helps with the whole section 10 (log monitoring) of PCI.

  • How much CPU does it use?

    • On the agent, it uses very little CPU/memory since it just read the events and forwards them to the manager.

    • On the manager, it depends on the number of events per second (EPS).

  • How does it deal with false positives?

    • False positives can be eliminated using local rules.

Configuration Options

These options should be specified locally in each agent’s ossec.conf file or the share agent.conf. Inside the <localfile> element, you can have the following options.

  • localfile

    • location

      Specify the location of the log to be read. strftime formats may be used for log file names. For instance, a log file named file.log-2011-01-22 could be referenced with file.log-%Y-%m-%d. Wildcards may be used on non-Windows systems. When wildcards are used, the log files must exist at the time ossec-logcollector is started. It will not automatically begin monitoring new log files. strftime and wildcards cannot be used on the same entry.

      Default: Multiple (eg /var/log/messages)

      Allowed: Any log file

    • log_format

      The format of the log being read.

      Note

      If the log has one entry per line, use syslog.

      Default: syslog

      Allowed:

      • syslog

        This format is for plain text files in a syslog-like format. It can also be used when there is no support for the logging format, and the logs are single line messages.

      • snort-full

        This is used for Snort’s full output format.

      • snort-fast

        This is used for Snort’s fast output format.

      • squid

      • iis

      • eventlog

        This is used for Microsoft Windows eventlog format.

      • eventchannel

        This is used for Microsoft Windows eventlogs, using the new EventApi. This allows OSSEC to monitor both standard “Windows” eventlogs and more recent “Application and Services” logs. This support was added in 2.8.

      Warning

      eventchannel cannot be used on Windows systems older than Vista.

      • mysql_log

        This is used for MySQL logs. It does not support multi-line logs.

      • postgresql_log

        This is used for PostgreSQL logs. It does not support multi-line logs.

      • nmapg

        This is used for monitoring files conforming to the grepable output from nmap.

      • apache

        This format is for apache’s default log format.

        Example:

        [Wed Jun  9 23:32:26 2010] [error] [client 192.168.1.100] File does not exist: /htdocs/favicon.ico

        Example:

        192.168.1.100 - - [21/Jan/2010:08:31:09 -0500] "GET / HTTP/1.0" 200 2212
      • command

        This format will be the output from the command (as run by root) defined by command. Each line of output will be treated as a separate log.

      • full_command

        This format will be the output from the command (as run by root) defined by command. The entire output will be treated as a single log.

      Warning

      command and full_command cannot be used in the agent.conf, and must be configured in each system’s ossec.conf.

      • djb-multilog

      • multi-line

        This option will allow applications that log multiple lines per event to be monitored. This format requires the number of lines to be consistent. multi-line: is followed by the number of lines in each log entry. Each line will be combined with the previous lines until all lines are gathered. There may be multiple timestamps in a finalized event.

        Allowed: <log_format>multi-line: NUMBER</log_format>

        Example:

        Log messages:

        Aug  9 14:22:47 hostname log line one
Aug  9 14:22:47 hostname log line two
Aug  9 14:22:47 hostname log line three
Aug  9 14:22:47 hostname log line four
Aug  9 14:22:47 hostname log line five

        Log message as analyzed by ossec-analysisd:

        Aug  9 14:22:47 hostname log line one Aug  9 14:22:47 hostname log line two Aug  9 14:22:47 hostname log line three Aug  9 14:22:47 hostname log line four Aug  9 14:22:47 hostname log line five
      • multi-line_indented

        This log format accepts logs spanning multiple lines with subsequent lines beginning with either a space or tab.

        Example:
        <log_format>multi-line_indented</log_format>

    • command

      The command to be run. All output from this command will be read as one or more log messages depending on whether command or full_command is used.

      Allowed: Any commandline and arguments.

    • alias

      An alias to identify the command. This will replace the command in the log message.

      For example <alias>usb-check</alias> would replace:

      ossec: output: 'reg QUERY HKLM\SYSTEM\CurrentControlSet\Enum\USBSTOR':

      with:

      ossec: output: 'usb-check':

      Allowed: Any string.

    • frequency

      The minimum time in seconds between command runs. The command will probably not run every frequency seconds exactly, but the time between runs will not be shorter than this setting. This is used with command and full_command.

      Allowed: Time in seconds.

    • check_diff

      The output from an event will be stored in an internal database. Every time the same event is received, the output is compared to the previous output. If the output has changed an alert will be generated.

    • only-future-events

      Only used with the eventchannel log format. By default, when OSSEC starts the eventchannel log format will read all events that ossec-logcollector missed since it was last stopped. It is possible to set only-future-events to yes in order to prevent this behaviour. When set to yes, OSSEC will only receive events that occured after the start of logcollector.

      <localfile>
  <location>System</location>
  <log_format>eventchannel</log_format>
  <only-future-events>yes</only-future-events>
</localfile>

    • query

      Only used with the eventchannel log format. It is possible to specify an XPATH query following the event schema (see Microsoft’s documentation) in order to filter the events that OSSEC will process.

      For example, the following configuration will only process events with an ID of 7040:

      <localfile>
  <location>System</location>
  <log_format>eventchannel</log_format>
  <query>Event/System[EventID=7040]</query>
</localfile>

Monitoring logs

With in OSSEC there are two major methods for monitoring logs: file and process. Each method has its own page and examples.