[ossec-list] Re: OSSEC Server Crashing on Solaris 9

["Daniel Cid" <daniel.cid@xxxxxxxxx>]

Hi Erik,

I have no clue of what is going on (well, I know that analysisd is
dying), but we can try to find it out.

Can you do the following (or all of them)?

-Start analysisd with the debug flags (-d -d)
-Run strace on analysisd before it dies (or something similar on
solaris -kdump?)

If we can't find out what is going on with it, it would be nice to
re-compile ossec
with debug enabled to see what is going on...

Thanks for the report,

--
Daniel B. Cid
dcid ( at ) ossec.net


On 6/10/07, Erik Delfgaauw <erik.delfgaauw@xxxxxxxxx> wrote:
> Hi folks,
>
> OSSEC Server is crashing after some time, it happens time after time, in
> this cycle which started on 2007/06/09 at 23:34:56 it happens on 2007/06/10
> at 02:55:35, here's some information:
>
> ==========[UNAME
> -A]====================================================================================================
>
> SunOS sola 5.9 Generic_118558-39 sun4u sparc SUNW,Sun-Blade-100 Solaris
>
> ==========[OSSEC.LOG]====================================================================================================
>
> 2007/06/09 23:34:56 ossec-maild: E-Mail notification disabled. Clean Exit.
> 2007/06/09 23:34:56 ossec-execd: Started (pid: 1689).
> 2007/06/09 23:34:56 ossec-analysisd: Reading rules file: 'rules_config.xml'
> 2007/06/09 23:34:56 ossec-analysisd: Reading rules file: 'pam_rules.xml'
> 2007/06/09 23:34:56 ossec-analysisd: Reading rules file: 'sshd_rules.xml'
> 2007/06/09 23:34:56 ossec-analysisd: Reading rules file: 'telnetd_rules.xml'
> 2007/06/09 23:34:56 ossec-analysisd: Reading rules file: 'syslog_rules.xml'
> 2007/06/09 23:34:56 ossec-analysisd: Reading rules file:
> 'arpwatch_rules.xml'
> 2007/06/09 23:34:56 ossec-analysisd: Reading rules file:
> 'symantec-av_rules.xml'
> 2007/06/09 23:34:56 ossec-analysisd: Reading rules file: 'pix_rules.xml'
> 2007/06/09 23:34:56 ossec-analysisd: Reading rules file: 'named_rules.xml'
> 2007/06/09 23:34:56 ossec-analysisd: Reading rules file: 'smbd_rules.xml'
> 2007/06/09 23:34:56 ossec-analysisd: Reading rules file: 'vsftpd_rules.xml'
> 2007/06/09 23:34:56 ossec-analysisd: Reading rules file:
> 'pure-ftpd_rules.xml'
> 2007/06/09 23:34:56 ossec-analysisd: Reading rules file: 'proftpd_rules.xml'
> 2007/06/09 23:34:57 ossec-analysisd: Reading rules file: 'ms_ftpd_rules.xml'
> 2007/06/09 23:34:57 ossec-analysisd: Reading rules file:
> 'hordeimp_rules.xml'
> 2007/06/09 23:34:57 ossec-analysisd: Reading rules file:
> 'vpopmail_rules.xml'
> 2007/06/09 23:34:57 ossec-analysisd: Reading rules file: 'web_rules.xml'
> 2007/06/09 23:34:57 ossec-analysisd: Reading rules file: 'apache_rules.xml'
> 2007/06/09 23:34:57 ossec-analysisd: Reading rules file: 'ids_rules.xml'
> 2007/06/09 23:34:57 ossec-analysisd: Reading rules file: 'squid_rules.xml'
> 2007/06/09 23:34:57 ossec-analysisd: Reading rules file:
> 'firewall_rules.xml'
> 2007/06/09 23:34:57 ossec-analysisd: Reading rules file:
> 'netscreenfw_rules.xml'
> 2007/06/09 23:34:57 ossec-analysisd: Reading rules file: 'postfix_rules.xml'
> 2007/06/09 23:34:57 ossec-analysisd: Reading rules file:
> 'sendmail_rules.xml'
> 2007/06/09 23:34:57 ossec-analysisd: Reading rules file: 'imapd_rules.xml'
> 2007/06/09 23:34:57 ossec-analysisd: Reading rules file:
> 'mailscanner_rules.xml'
> 2007/06/09 23:34:57 ossec-analysisd: Reading rules file:
> 'ms-exchange_rules.xml'
> 2007/06/09 23:34:57 ossec-analysisd: Reading rules file: 'racoon_rules.xml'
> 2007/06/09 23:34:57 ossec-analysisd: Reading rules file:
> 'vpn_concentrator_rules.xml'
> 2007/06/09 23:34:57 ossec-analysisd: Reading rules file: 'spamd_rules.xml'
> 2007/06/09 23:34:57 ossec-analysisd: Reading rules file: 'msauth_rules.xml'
> 2007/06/09 23:34:57 ossec-analysisd: Reading rules file: 'attack_rules.xml'
> 2007/06/09 23:34:57 ossec-analysisd: Reading rules file: 'zeus_rules.xml'
> 2007/06/09 23:34:57 ossec-analysisd: Reading rules file: 'ossec_rules.xml'
> 2007/06/09 23:34:57 ossec-analysisd: Reading rules file: 'local_rules.xml'
> 2007/06/09 23:34:57 ossec-analysisd: Total rules enabled: '559'
> 2007/06/09 23:34:57 ossec-analysisd: Ignoring file: '/etc/mtab'
> 2007/06/09 23:34:57 ossec-analysisd: Ignoring file: '/etc/mnttab'
> 2007/06/09 23:34:57 ossec-analysisd: Ignoring file: '/etc/hosts.deny'
> 2007/06/09 23:34:57 ossec-analysisd: Ignoring file: '/etc/mail/statistics'
> 2007/06/09 23:34:57 ossec-analysisd: Ignoring file: '/etc/random-seed'
> 2007/06/09 23:34:57 ossec-analysisd: Ignoring file: '/etc/adjtime'
> 2007/06/09 23:34:57 ossec-analysisd: Ignoring file: '/etc/httpd/logs'
> 2007/06/09 23:34:57 ossec-analysisd: Ignoring file: '/etc/utmpx'
> 2007/06/09 23:34:57 ossec-analysisd: Ignoring file: '/etc/wtmpx'
> 2007/06/09 23:34:57 ossec-analysisd: Ignoring file: '/etc/cups/certs'
> 2007/06/09 23:34:57 ossec-analysisd: Ignoring file:
> 'C:\WINDOWS/System32/LogFiles'
> 2007/06/09 23:34:57 ossec-analysisd: Ignoring file: 'C:\WINDOWS/Debug'
> 2007/06/09 23:34:57 ossec-analysisd: Ignoring file:
> 'C:\WINDOWS/WindowsUpdate.log'
> 2007/06/09 23:34:57 ossec-analysisd: Ignoring file: 'C:\WINDOWS/iis6.log'
> 2007/06/09 23:34:57 ossec-analysisd: Ignoring file:
> 'C:\WINDOWS/system32/wbem/Logs'
> 2007/06/09 23:34:57 ossec-analysisd: Ignoring file:
> 'C:\WINDOWS/system32/wbem/Repository'
> 2007/06/09 23:34:57 ossec-analysisd: Ignoring file: 'C:\WINDOWS/Prefetch'
> 2007/06/09 23:34:57 ossec-analysisd: Ignoring file:
> 'C:\WINDOWS/PCHEALTH/HELPCTR/DataColl'
> 2007/06/09 23:34:57 ossec-analysisd: Ignoring file:
> 'C:\WINDOWS/SoftwareDistribution'
> 2007/06/09 23:34:57 ossec-analysisd: Ignoring file: 'C:\WINDOWS/Temp'
> 2007/06/09 23:34:57 ossec-analysisd: Ignoring file:
> 'C:\WINDOWS/system32/config'
> 2007/06/09 23:34:57 ossec-analysisd: Ignoring file:
> 'C:\WINDOWS/system32/spool'
> 2007/06/09 23:34:57 ossec-analysisd: Ignoring file:
> 'C:\WINDOWS/system32/CatRoot'
> 2007/06/09 23:34:57 ossec-analysisd: White listing IP: '127.0.0.1'
> 2007/06/09 23:34:57 ossec-analysisd: White listing IP: '10.6.1.250'
>  2007/06/09 23:34:57 ossec-analysisd: 2 IPs in the white list for active
> response.
> 2007/06/09 23:34:57 ossec-analysisd: White listing Hostname:
> 'localhost.localdomain'
> 2007/06/09 23:34:57 ossec-analysisd: 1 Hostname(s) in the white list for
> active response.
> 2007/06/09 23:34:57 ossec-analysisd: Started (pid: 1694).
> 2007/06/09 23:34:57 ossec-logcollector: DEBUG: Waiting main daemons to
> settle.
> 2007/06/09 23:34:57 ossec-remoted: Started (pid: 1702).
> 2007/06/09 23:34:57 ossec-remoted: Started (pid: 1703).
> 2007/06/09 23:34:57 ossec-remoted: Assigning counter for agent cent:
> '1:8010'.
> 2007/06/09 23:34:57 ossec-remoted: Assigning counter for agent wall:
> '1:4126'.
> 2007/06/09 23:34:57 ossec-remoted: Assigning sender counter: 0:199
> 2007/06/09 23:34:57 ossec-monitord: Started (pid: 1711).
> 2007/06/09 23:34:59 ossec-syscheckd: Started (pid: 1706).
> 2007/06/09 23:34:59 ossec-rootcheck: Started (pid: 1706).
> 2007/06/09 23:35:00 ossec-analysisd: Connected to '/queue/alerts/ar'
> (active-response queue)
> 2007/06/09 23:35:00 ossec-analysisd: Connected to '/queue/alerts/execq'
> (exec queue)
> 2007/06/09 23:35:03 ossec-logcollector: (unix_domain) Maximum send buffer
> set to: '16384'.
> 2007/06/09 23:35:03 ossec-logcollector: DEBUG: Entering LogCollectorStart().
> 2007/06/09 23:35:03 ossec-logcollector(1950): Analyzing file:
> '/var/log/authlog'.
> 2007/06/09 23:35:03 ossec-logcollector(1950): Analyzing file:
> '/var/log/syslog'.
> 2007/06/09 23:35:03 ossec-logcollector: Started (pid: 1698).
> 2007/06/10 01:03:46 ossec-logcollector: DEBUG: Reading syslog message: 'Jun
> 10 01:03:44 sola EEPROM_SECURITY: [ID 702911 auth.info]
> security-#badlogins=0'
> 2007/06/10 01:11:25 ossec-logcollector: DEBUG: Reading syslog message: 'Jun
> 10 01:11:23 sola syslogd: configuration restart'
> 2007/06/10 01:12:15 ossec-logcollector: DEBUG: Reading syslog message: 'Jun
> 10 01:12:14 sola syslogd: going down on signal 15'
> 2007/06/10 02:55:35 ossec-logcollector: DEBUG: Reading syslog message: 'Jun
> 10 02:55:33 sola genunix: [ID 457380 kern.notice] NOTICE: core_log:
> ossec-analysisd[1694] core dump failed, errno=2:
> /var/core/core_sola_ossec-analysisd_201_201_1181436933_1694'
> 2007/06/10 02:55:35 ossec-logcollector: socketerr (not available).
> 2007/06/10 02:55:35 ossec-logcollector(1224): Error sending message to
> queue.
> 2007/06/10 02:55:38 ossec-logcollector(1210): Queue
> '/opt/ossec/queue/ossec/queue' not accessible: 'Destination address
> required'.
> 2007/06/10 02:55:38 ossec-logcollector(1211): Unable to access queue:
> '/opt/ossec/queue/ossec/queue'. Giving up..
> 2007/06/10 02:55:44 ossec-remoted: socketerr (not available).
> 2007/06/10 03:19:17 ossec-monitord: socketerr (not available).
> 2007/06/10 03:19:17 ossec-monitord(1224): Error sending message to queue.
> 2007/06/10 03:19:17 ossec-monitord: socketerr (not available).
> 2007/06/10 03:19:17 ossec-monitord(1224): Error sending message to queue.
> 2007/06/10 06:55:53 ossec-syscheckd: socketerr (not available).
> 2007/06/10 06:55:53 ossec-syscheckd(1224): Error sending message to queue.
> 2007/06/10 06:55:56 ossec-syscheckd(1210): Queue
> '/opt/ossec/queue/ossec/queue' not accessible: 'Destination address
> required'.
> 2007/06/10 06:55:56 ossec-syscheckd(1211): Unable to access queue:
> '/opt/ossec/queue/ossec/queue'. Giving up..
>
> ==========[OSSEC-INIT.CONF]====================================================================================================
>
> DIRECTORY="/opt/ossec"
> VERSION="v1.2"
> DATE="Sun May 20 00:43:09 MEST 2007"
> TYPE="server"
>
> ==========[OSSEC.CONF]====================================================================================================
>
> <ossec_config>
>   <global>
>     <email_notification>no</email_notification>
>   </global>
>
>   <rules>
>     <include>rules_config.xml</include>
>     <include>pam_rules.xml</include>
>     <include>sshd_rules.xml</include>
>     <include>telnetd_rules.xml</include>
>     <include>syslog_rules.xml</include>
>     <include>arpwatch_rules.xml</include>
>     <include>symantec-av_rules.xml</include>
>     <include>pix_rules.xml</include>
>     <include>named_rules.xml</include>
>     <include>smbd_rules.xml</include>
>     <include>vsftpd_rules.xml</include>
>     <include>pure-ftpd_rules.xml</include>
>      <include>proftpd_rules.xml</include>
>     <include>ms_ftpd_rules.xml</include>
>     <include>hordeimp_rules.xml</include>
>     <include>vpopmail_rules.xml</include>
>     <include>web_rules.xml</include>
>     <include>apache_rules.xml</include>
>     <include>ids_rules.xml</include>
>     <include>squid_rules.xml</include>
>     <include>firewall_rules.xml</include>
>     <include>netscreenfw_rules.xml</include>
>     <include>postfix_rules.xml</include>
>     <include>sendmail_rules.xml</include>
>     <include>imapd_rules.xml</include>
>     <include>mailscanner_rules.xml</include>
>     <include>ms-exchange_rules.xml</include>
>     <include>racoon_rules.xml</include>
>     <include>vpn_concentrator_rules.xml</include>
>     <include>spamd_rules.xml</include>
>     <include>msauth_rules.xml</include>
>     <!-- <include>policy_rules.xml</include> -->
>     <include>attack_rules.xml</include>
>     <include>zeus_rules.xml</include>
>     <include>ossec_rules.xml</include>
>     <include>local_rules.xml</include>
>   </rules>
>
>   <syscheck>
>     <!-- Frequency that syscheck is executed - default to every 6 hours -->
>     <frequency>21600</frequency>
>
>     <!-- Directories to check  (perform all possible verifications) -->
>     <directories
> check_all="yes">/etc,/usr/bin,/usr/sbin</directories>
>     <directories check_all="yes">/bin,/sbin</directories>
>
>     <!-- Files/directories to ignore -->
>     <ignore>/etc/mtab</ignore>
>     <ignore>/etc/mnttab</ignore>
>     <ignore>/etc/hosts.deny</ignore>
>     <ignore>/etc/mail/statistics</ignore>
>     <ignore>/etc/random-seed</ignore>
>     <ignore>/etc/adjtime</ignore>
>     <ignore>/etc/httpd/logs</ignore>
>     <ignore>/etc/utmpx</ignore>
>     <ignore>/etc/wtmpx</ignore>
>     <ignore>/etc/cups/certs</ignore>
>
>     <!-- Windows files to ignore -->
>     <ignore>C:\WINDOWS/System32/LogFiles</ignore>
>     <ignore>C:\WINDOWS/Debug</ignore>
>     <ignore>C:\WINDOWS/WindowsUpdate.log</ignore>
>     <ignore>C:\WINDOWS/iis6.log</ignore>
>     <ignore>C:\WINDOWS/system32/wbem/Logs</ignore>
>     <ignore>C:\WINDOWS/system32/wbem/Repository</ignore>
>     <ignore>C:\WINDOWS/Prefetch</ignore>
>     <ignore>C:\WINDOWS/PCHEALTH/HELPCTR/DataColl</ignore>
>     <ignore>C:\WINDOWS/SoftwareDistribution</ignore>
>     <ignore>C:\WINDOWS/Temp</ignore>
>     <ignore>C:\WINDOWS/system32/config</ignore>
>     <ignore>C:\WINDOWS/system32/spool</ignore>
>     <ignore>C:\WINDOWS/system32/CatRoot</ignore>
>   </syscheck>
>
>   <rootcheck>
>
> <rootkit_files>/opt/ossec/etc/shared/rootkit_files.txt</rootkit_files>
>
> <rootkit_trojans>/opt/ossec/etc/shared/rootkit_trojans.txt</rootkit_trojans>
>   </rootcheck>
>
>   <global>
>     <white_list>127.0.0.1</white_list>
>     <white_list>^localhost.localdomain$</white_list>
>     <white_list> 10.6.1.250</white_list>
>   </global>
>
>   <remote>
>     <connection>secure</connection>
>   </remote>
>
>   <alerts>
>     <log_alert_level>1</log_alert_level>
>   </alerts>
>
>   <command>
>     <name>host-deny</name>
>     <executable>host-deny.sh</executable>
>     <expect>srcip</expect>
>     <timeout_allowed>yes</timeout_allowed>
>   </command>
>
>   <command>
>     <name>firewall-drop</name>
>     <executable>firewall-drop.sh</executable>
>     <expect>srcip</expect>
>     <timeout_allowed>yes</timeout_allowed>
>   </command>
>
>   <command>
>     <name>disable-account</name>
>     <executable>disable-account.sh</executable>
>     <expect>user</expect>
>     <timeout_allowed>yes</timeout_allowed>
>   </command>
>
>   <command>
>     <name>route-null</name>
>     <executable>route-null.sh</executable>
>     <expect>srcip</expect>
>     <timeout_allowed>yes</timeout_allowed>
>   </command>
>
>
>   <!-- Active Response Config -->
>   <active-response>
>     <!-- This response is going to execute the host-deny
>        - command for every event that fires a rule with
>        - level (severity) >= 6.
>        - The IP is going to be blocked for  600 seconds.
>       -->
>     <command>host-deny</command>
>     <location>local</location>
>     <level>6</level>
>     <timeout>600</timeout>
>   </active-response>
>
>   <active-response>
>     <!-- Firewall Drop response. Block the IP for
>        - 600 seconds on the firewall (iptables,
>        - ipfilter, etc).
>       -->
>     <command>firewall-drop</command>
>     <location>local</location>
>     <level>6</level>
>     <timeout>600</timeout>
>   </active-response>
>
>   <!-- Files to monitor (localfiles) -->
>
>   <localfile>
>     <log_format>syslog</log_format>
>     <location>/var/log/authlog</location>
>   </localfile>
>
>    <localfile>
>     <log_format>syslog</log_format>
>     <location>/var/log/syslog</location>
>   </localfile>
> </ossec_config>
>
>
> Hope you can get a clue from all this.
>
> Many thanks!
>
> Erik
>