[ossec-list] Re: OSSEC Server Crashing on Solaris 9

    # We actually start them now.
    for i in ${SDAEMONS}; do
        pstatus ${i};
        if [ $? = 0 ]; then
            ${DIR}/bin/${i} -d -d;
            if [ $? != 0 ]; then
                unlock;
                exit 1;
            fi

            echo "Started ${i}..."
        else
            echo "${i} already running..."
        fi

    done

==================================================

I then start OSSEC using ./ossec-control start in /opt/ossec/bin, which outputs the following:

==================================================

Starting OSSEC HIDS v1.2 (by Daniel B. Cid)...
2007/06/17 16:38:16 ossec-maild: Starting ...
2007/06/17 16:38:16 ossec-maild: E-Mail notification disabled. Clean Exit.
Started ossec-maild...
Started ossec-execd...
2007/06/17 16:38:16 ossec-analysisd: Starting ...
2007/06/17 16:38:16 ossec-analysisd: Found user/group ...
2007/06/17 16:38:16 ossec-analysisd: Active response initialized ...
2007/06/17 16:38:16 ossec-analysisd: Read configuration ...
Started ossec-analysisd...
2007/06/17 16:38:16 ossec-logcollector: Starting ...
Started ossec-logcollector...
2007/06/17 16:38:17 ossec-remoted: Starting ...
Started ossec-remoted...
2007/06/17 16:38:17 ossec-rootcheck: Starting ...
2007/06/17 16:38:17 ossec-rootcheck: Starting queue ...
2007/06/17 16:38:20 ossec-syscheckd(1210): Queue '/opt/ossec/queue/ossec/queue' not accessible: 'Destination address required'.
2007/06/17 16:38:20 ossec-rootcheck(1210): Queue '/opt/ossec/queue/ossec/queue' not accessible: 'Destination address required'.
2007/06/17 16:38:28 ossec-syscheckd(1210): Queue '/opt/ossec/queue/ossec/queue' not accessible: 'Destination address required'.
2007/06/17 16:38:28 ossec-rootcheck(1210): Queue '/opt/ossec/queue/ossec/queue' not accessible: 'Destination address required'.
2007/06/17 16:38:41 ossec-syscheckd(1210): Queue '/opt/ossec/queue/ossec/queue' not accessible: 'Destination address required'.
2007/06/17 16:38:41 ossec-rootcheck(1211): Unable to access queue: '/opt/ossec/queue/ossec/queue'. Giving up..

==================================================

The OSSEC log file then contains the following:

==================================================

2007/06/17 16:38:16 ossec-maild: Starting ...
2007/06/17 16:38:16 ossec-maild: E-Mail notification disabled. Clean Exit.
2007/06/17 16:38:16 ossec-execd: Started (pid: 10759).
2007/06/17 16:38:16 ossec-analysisd: Starting ...
2007/06/17 16:38:16 ossec-analysisd: Found user/group ...
2007/06/17 16:38:16 ossec-analysisd: Active response initialized ...
2007/06/17 16:38:16 ossec-analysisd: Read configuration ...
2007/06/17 16:38:16 ossec-logcollector: Starting ...
2007/06/17 16:38:17 ossec-logcollector: DEBUG: Waiting main daemons to settle.
2007/06/17 16:38:17 ossec-remoted: Starting ...
2007/06/17 16:38:17 ossec-remoted: Started (pid: 10770).
2007/06/17 16:38:17 ossec-remoted: DEBUG: Forking remoted: '0'.
2007/06/17 16:38:17 ossec-remoted: Started (pid: 10771).
2007/06/17 16:38:17 ossec-remoted: DEBUG: Starting manager_unit
2007/06/17 16:38:17 ossec-rootcheck: Starting ...
2007/06/17 16:38:17 ossec-rootcheck: Starting queue ...
2007/06/17 16:38:20 ossec-remoted(1210): Queue '/queue/ossec/queue' not accessible: 'Destination address required'.
2007/06/17 16:38:20 ossec-remoted(1211): Unable to access queue: '/queue/ossec/queue'. Giving up..
2007/06/17 16:38:20 ossec-syscheckd(1210): Queue '/opt/ossec/queue/ossec/queue' not accessible: 'Destination address required'.
2007/06/17 16:38:20 ossec-rootcheck(1210): Queue '/opt/ossec/queue/ossec/queue' not accessible: 'Destination address required'.
2007/06/17 16:38:26 ossec-logcollector(1210): Queue '/opt/ossec/queue/ossec/queue' not accessible: 'Destination address required'.
2007/06/17 16:38:26 ossec-logcollector(1211): Unable to access queue: '/opt/ossec/queue/ossec/queue'. Giving up..
2007/06/17 16:38:28 ossec-syscheckd(1210): Queue '/opt/ossec/queue/ossec/queue' not accessible: 'Destination address required'.
2007/06/17 16:38:28 ossec-rootcheck(1210): Queue '/opt/ossec/queue/ossec/queue' not accessible: 'Destination address required'.
2007/06/17 16:38:41 ossec-syscheckd(1210): Queue '/opt/ossec/queue/ossec/queue' not accessible: 'Destination address required'.
2007/06/17 16:38:41 ossec-rootcheck(1211): Unable to access queue: '/opt/ossec/queue/ossec/queue'. Giving up..

==================================================

If I leave out "-d -d", OSSEC shows the following when starting:

==================================================

Starting OSSEC HIDS v1.2 (by Daniel B. Cid)...
2007/06/17 16:40:32 ossec-maild: E-Mail notification disabled. Clean Exit.
Started ossec-maild...
Started ossec-execd...
Started ossec-analysisd...
Started ossec-logcollector...
Started ossec-remoted...
Started ossec-syscheckd...
Started ossec-monitord...
Completed.

==================================================

The OSSEC log file contains nothing out of the ordinary with "-d -d" omitted.

This is how the access rights are set of /opt/ossec/queue/ossec/queue (default, never changed it):

==================================================

0 srw-rw---- 1 ossec ossec 0 Jun 17 16:40 /opt/ossec/queue/ossec/queue

==================================================

Any clue now? :-)

If not I'll proceed with trussing (stracing) the OSSEC startup.

What I am wondering, can it have anything to do with the fact that we use umask 0022 on our systems? Remember that I'm also still struggling with OSSEC-WUI? Perhaps a script that sets all rights for OSSEC-WUI and OSSEC-HIDS (latest versions) will help us further? I've seen one on the Wiki site, but it gives errors and made me wonder if the script is up to date.

Anyway, curious as hell, I hope to hear from you, cheers!

E.

2007/6/15, Daniel Cid <daniel.cid@xxxxxxxxx>:

Hi Erik,

I have no clue of what is going on (well, I know that analysisd is
dying), but we can try to find it out.

Can you do the following (or all of them)?

-Start analysisd with the debug flags (-d -d)
-Run strace on analysisd before it dies (or something similar on
solaris -kdump?)

If we can't find out what is going on with it, it would be nice to
re-compile ossec
with debug enabled to see what is going on...

Thanks for the report,

--
Daniel B. Cid
dcid ( at ) ossec.net

On 6/10/07, Erik Delfgaauw < erik.delfgaauw@xxxxxxxxx> wrote:
> Hi folks,
>
> OSSEC Server is crashing after some time, it happens time after time, in
> this cycle which started on 2007/06/09 at 23:34:56 it happens on 2007/06/10
> at 02:55:35, here's some information:
>
> ==========[UNAME
> -A]====================================================================================================
>
> SunOS sola 5.9 Generic_118558-39 sun4u sparc SUNW,Sun-Blade-100 Solaris
>
> ==========[OSSEC.LOG]====================================================================================================
>
> 2007/06/09 23:34:56 ossec-maild: E-Mail notification disabled. Clean Exit.
> 2007/06/09 23:34:56 ossec-execd: Started (pid: 1689).
> 2007/06/09 23:34:56 ossec-analysisd: Reading rules file: 'rules_config.xml'
> 2007/06/09 23:34:56 ossec-analysisd: Reading rules file: 'pam_rules.xml'
> 2007/06/09 23:34:56 ossec-analysisd: Reading rules file: 'sshd_rules.xml'
> 2007/06/09 23:34:56 ossec-analysisd: Reading rules file: 'telnetd_rules.xml'
> 2007/06/09 23:34:56 ossec-analysisd: Reading rules file: 'syslog_rules.xml'
> 2007/06/09 23:34:56 ossec-analysisd: Reading rules file:
> 'arpwatch_rules.xml'
> 2007/06/09 23:34:56 ossec-analysisd: Reading rules file:
> 'symantec-av_rules.xml'
> 2007/06/09 23:34:56 ossec-analysisd: Reading rules file: 'pix_rules.xml'
> 2007/06/09 23:34:56 ossec-analysisd: Reading rules file: 'named_rules.xml'
> 2007/06/09 23:34:56 ossec-analysisd: Reading rules file: 'smbd_rules.xml'
> 2007/06/09 23:34:56 ossec-analysisd: Reading rules file: 'vsftpd_rules.xml'
> 2007/06/09 23:34:56 ossec-analysisd: Reading rules file:
> 'pure-ftpd_rules.xml'
> 2007/06/09 23:34:56 ossec-analysisd: Reading rules file: 'proftpd_rules.xml'
> 2007/06/09 23:34:57 ossec-analysisd: Reading rules file: 'ms_ftpd_rules.xml'
> 2007/06/09 23:34:57 ossec-analysisd: Reading rules file:
> 'hordeimp_rules.xml'
> 2007/06/09 23:34:57 ossec-analysisd: Reading rules file:
> 'vpopmail_rules.xml'
> 2007/06/09 23:34:57 ossec-analysisd: Reading rules file: 'web_rules.xml'
> 2007/06/09 23:34:57 ossec-analysisd: Reading rules file: 'apache_rules.xml'
> 2007/06/09 23:34:57 ossec-analysisd: Reading rules file: 'ids_rules.xml'
> 2007/06/09 23:34:57 ossec-analysisd: Reading rules file: 'squid_rules.xml'
> 2007/06/09 23:34:57 ossec-analysisd: Reading rules file:
> 'firewall_rules.xml'
> 2007/06/09 23:34:57 ossec-analysisd: Reading rules file:
> 'netscreenfw_rules.xml'
> 2007/06/09 23:34:57 ossec-analysisd: Reading rules file: 'postfix_rules.xml'
> 2007/06/09 23:34:57 ossec-analysisd: Reading rules file:
> 'sendmail_rules.xml'
> 2007/06/09 23:34:57 ossec-analysisd: Reading rules file: 'imapd_rules.xml'
> 2007/06/09 23:34:57 ossec-analysisd: Reading rules file:
> 'mailscanner_rules.xml'
> 2007/06/09 23:34:57 ossec-analysisd: Reading rules file:
> 'ms-exchange_rules.xml'
> 2007/06/09 23:34:57 ossec-analysisd: Reading rules file: 'racoon_rules.xml'
> 2007/06/09 23:34:57 ossec-analysisd: Reading rules file:
> 'vpn_concentrator_rules.xml'
> 2007/06/09 23:34:57 ossec-analysisd: Reading rules file: 'spamd_rules.xml'
> 2007/06/09 23:34:57 ossec-analysisd: Reading rules file: 'msauth_rules.xml'
> 2007/06/09 23:34:57 ossec-analysisd: Reading rules file: 'attack_rules.xml'
> 2007/06/09 23:34:57 ossec-analysisd: Reading rules file: 'zeus_rules.xml'
> 2007/06/09 23:34:57 ossec-analysisd: Reading rules file: 'ossec_rules.xml'
> 2007/06/09 23:34:57 ossec-analysisd: Reading rules file: 'local_rules.xml'
> 2007/06/09 23:34:57 ossec-analysisd: Total rules enabled: '559'
> 2007/06/09 23:34:57 ossec-analysisd: Ignoring file: '/etc/mtab'
> 2007/06/09 23:34:57 ossec-analysisd: Ignoring file: '/etc/mnttab'
> 2007/06/09 23:34:57 ossec-analysisd: Ignoring file: '/etc/hosts.deny'
> 2007/06/09 23:34:57 ossec-analysisd: Ignoring file: '/etc/mail/statistics'
> 2007/06/09 23:34:57 ossec-analysisd: Ignoring file: '/etc/random-seed'
> 2007/06/09 23:34:57 ossec-analysisd: Ignoring file: '/etc/adjtime'
> 2007/06/09 23:34:57 ossec-analysisd: Ignoring file: '/etc/httpd/logs'
> 2007/06/09 23:34:57 ossec-analysisd: Ignoring file: '/etc/utmpx'
> 2007/06/09 23:34:57 ossec-analysisd: Ignoring file: '/etc/wtmpx'
> 2007/06/09 23:34:57 ossec-analysisd: Ignoring file: '/etc/cups/certs'
> 2007/06/09 23:34:57 ossec-analysisd: Ignoring file:
> 'C:\WINDOWS/System32/LogFiles'
> 2007/06/09 23:34:57 ossec-analysisd: Ignoring file: 'C:\WINDOWS/Debug'
> 2007/06/09 23:34:57 ossec-analysisd: Ignoring file:
> 'C:\WINDOWS/WindowsUpdate.log'
> 2007/06/09 23:34:57 ossec-analysisd: Ignoring file: 'C:\WINDOWS/iis6.log'
> 2007/06/09 23:34:57 ossec-analysisd: Ignoring file:
> 'C:\WINDOWS/system32/wbem/Logs'
> 2007/06/09 23:34:57 ossec-analysisd: Ignoring file:
> 'C:\WINDOWS/system32/wbem/Repository'
> 2007/06/09 23:34:57 ossec-analysisd: Ignoring file: 'C:\WINDOWS/Prefetch'
> 2007/06/09 23:34:57 ossec-analysisd: Ignoring file:
> 'C:\WINDOWS/PCHEALTH/HELPCTR/DataColl'
> 2007/06/09 23:34:57 ossec-analysisd: Ignoring file:
> 'C:\WINDOWS/SoftwareDistribution'
> 2007/06/09 23:34:57 ossec-analysisd: Ignoring file: 'C:\WINDOWS/Temp'
> 2007/06/09 23:34:57 ossec-analysisd: Ignoring file:
> 'C:\WINDOWS/system32/config'
> 2007/06/09 23:34:57 ossec-analysisd: Ignoring file:
> 'C:\WINDOWS/system32/spool'
> 2007/06/09 23:34:57 ossec-analysisd: Ignoring file:
> 'C:\WINDOWS/system32/CatRoot'
> 2007/06/09 23:34:57 ossec-analysisd: White listing IP: '127.0.0.1'
> 2007/06/09 23:34:57 ossec-analysisd: White listing IP: ' 10.6.1.250'
>  2007/06/09 23:34:57 ossec-analysisd: 2 IPs in the white list for active
> response.
> 2007/06/09 23:34:57 ossec-analysisd: White listing Hostname:
> ' localhost.localdomain'
> 2007/06/09 23:34:57 ossec-analysisd: 1 Hostname(s) in the white list for
> active response.
> 2007/06/09 23:34:57 ossec-analysisd: Started (pid: 1694).
> 2007/06/09 23:34:57 ossec-logcollector: DEBUG: Waiting main daemons to
> settle.
> 2007/06/09 23:34:57 ossec-remoted: Started (pid: 1702).
> 2007/06/09 23:34:57 ossec-remoted: Started (pid: 1703).
> 2007/06/09 23:34:57 ossec-remoted: Assigning counter for agent cent:
> '1:8010'.
> 2007/06/09 23:34:57 ossec-remoted: Assigning counter for agent wall:
> '1:4126'.
> 2007/06/09 23:34:57 ossec-remoted: Assigning sender counter: 0:199
> 2007/06/09 23:34:57 ossec-monitord: Started (pid: 1711).
> 2007/06/09 23:34:59 ossec-syscheckd: Started (pid: 1706).
> 2007/06/09 23:34:59 ossec-rootcheck: Started (pid: 1706).
> 2007/06/09 23:35:00 ossec-analysisd: Connected to '/queue/alerts/ar'
> (active-response queue)
> 2007/06/09 23:35:00 ossec-analysisd: Connected to '/queue/alerts/execq'
> (exec queue)
> 2007/06/09 23:35:03 ossec-logcollector: (unix_domain) Maximum send buffer
> set to: '16384'.
> 2007/06/09 23:35:03 ossec-logcollector: DEBUG: Entering LogCollectorStart().
> 2007/06/09 23:35:03 ossec-logcollector(1950): Analyzing file:
> '/var/log/authlog'.
> 2007/06/09 23:35:03 ossec-logcollector(1950): Analyzing file:
> '/var/log/syslog'.
> 2007/06/09 23:35:03 ossec-logcollector: Started (pid: 1698).
> 2007/06/10 01:03:46 ossec-logcollector: DEBUG: Reading syslog message: 'Jun
> 10 01:03:44 sola EEPROM_SECURITY: [ID 702911 auth.info]
> security-#badlogins=0'
> 2007/06/10 01:11:25 ossec-logcollector: DEBUG: Reading syslog message: 'Jun
> 10 01:11:23 sola syslogd: configuration restart'
> 2007/06/10 01:12:15 ossec-logcollector: DEBUG: Reading syslog message: 'Jun
> 10 01:12:14 sola syslogd: going down on signal 15'
> 2007/06/10 02:55:35 ossec-logcollector: DEBUG: Reading syslog message: 'Jun
> 10 02:55:33 sola genunix: [ID 457380 kern.notice] NOTICE: core_log:
> ossec-analysisd[1694] core dump failed, errno=2:
> /var/core/core_sola_ossec-analysisd_201_201_1181436933_1694'
> 2007/06/10 02:55:35 ossec-logcollector: socketerr (not available).
> 2007/06/10 02:55:35 ossec-logcollector(1224): Error sending message to
> queue.
> 2007/06/10 02:55:38 ossec-logcollector(1210): Queue
> '/opt/ossec/queue/ossec/queue' not accessible: 'Destination address
> required'.
> 2007/06/10 02:55:38 ossec-logcollector(1211): Unable to access queue:
> '/opt/ossec/queue/ossec/queue'. Giving up..
> 2007/06/10 02:55:44 ossec-remoted: socketerr (not available).
> 2007/06/10 03:19:17 ossec-monitord: socketerr (not available).
> 2007/06/10 03:19:17 ossec-monitord(1224): Error sending message to queue.
> 2007/06/10 03:19:17 ossec-monitord: socketerr (not available).
> 2007/06/10 03:19:17 ossec-monitord(1224): Error sending message to queue.
> 2007/06/10 06:55:53 ossec-syscheckd: socketerr (not available).
> 2007/06/10 06:55:53 ossec-syscheckd(1224): Error sending message to queue.
> 2007/06/10 06:55:56 ossec-syscheckd(1210): Queue
> '/opt/ossec/queue/ossec/queue' not accessible: 'Destination address
> required'.
> 2007/06/10 06:55:56 ossec-syscheckd(1211): Unable to access queue:
> '/opt/ossec/queue/ossec/queue'. Giving up..
>
> ==========[OSSEC-INIT.CONF]====================================================================================================
>
> DIRECTORY="/opt/ossec"
> VERSION="v1.2"
> DATE="Sun May 20 00:43:09 MEST 2007"
> TYPE="server"
>
> ==========[OSSEC.CONF]====================================================================================================
>
> <ossec_config>
>   <global>
>     <email_notification>no</email_notification>
>   </global>
>
>   <rules>
>     <include>rules_config.xml</include>
>     <include>pam_rules.xml</include>
>     <include>sshd_rules.xml</include>
>     <include>telnetd_rules.xml</include>
>     <include>syslog_rules.xml</include>
>     <include>arpwatch_rules.xml</include>
>     <include>symantec-av_rules.xml</include>
>     <include>pix_rules.xml</include>
>     <include>named_rules.xml</include>
>     <include>smbd_rules.xml</include>
>     <include>vsftpd_rules.xml</include>
>     <include>pure-ftpd_rules.xml</include>
>      <include>proftpd_rules.xml</include>
>     <include>ms_ftpd_rules.xml</include>
>     <include>hordeimp_rules.xml</include>
>     <include>vpopmail_rules.xml</include>
>     <include>web_rules.xml</include>
>     <include>apache_rules.xml</include>
>     <include>ids_rules.xml</include>
>     <include>squid_rules.xml</include>
>     <include>firewall_rules.xml</include>
>     <include>netscreenfw_rules.xml</include>
>     <include>postfix_rules.xml</include>
>     <include>sendmail_rules.xml</include>
>     <include>imapd_rules.xml</include>
>     <include>mailscanner_rules.xml</include>
>     <include>ms-exchange_rules.xml</include>
>     <include>racoon_rules.xml</include>
>     <include>vpn_concentrator_rules.xml</include>
>     <include>spamd_rules.xml</include>
>     <include>msauth_rules.xml</include>
>     
>     <include>attack_rules.xml</include>
>     <include>zeus_rules.xml</include>
>     <include>ossec_rules.xml</include>
>     <include>local_rules.xml</include>
>   </rules>
>
>   <syscheck>
>     
>     <frequency>21600</frequency>
>
>     
>     <directories
> check_all="yes">/etc,/usr/bin,/usr/sbin</directories>
>     <directories check_all="yes">/bin,/sbin</directories>
>
>     
>     <ignore>/etc/mtab</ignore>
>     <ignore>/etc/mnttab</ignore>
>     <ignore>/etc/hosts.deny</ignore>
>     <ignore>/etc/mail/statistics</ignore>
>     <ignore>/etc/random-seed</ignore>
>     <ignore>/etc/adjtime</ignore>
>     <ignore>/etc/httpd/logs</ignore>
>     <ignore>/etc/utmpx</ignore>
>     <ignore>/etc/wtmpx</ignore>
>     <ignore>/etc/cups/certs</ignore>
>
>     
>     <ignore>C:\WINDOWS/System32/LogFiles</ignore>
>     <ignore>C:\WINDOWS/Debug</ignore>
>     <ignore>C:\WINDOWS/WindowsUpdate.log</ignore>
>     <ignore>C:\WINDOWS/iis6.log</ignore>
>     <ignore>C:\WINDOWS/system32/wbem/Logs</ignore>
>     <ignore>C:\WINDOWS/system32/wbem/Repository</ignore>
>     <ignore>C:\WINDOWS/Prefetch</ignore>
>     <ignore>C:\WINDOWS/PCHEALTH/HELPCTR/DataColl</ignore>
>     <ignore>C:\WINDOWS/SoftwareDistribution</ignore>
>     <ignore>C:\WINDOWS/Temp</ignore>
>     <ignore>C:\WINDOWS/system32/config</ignore>
>     <ignore>C:\WINDOWS/system32/spool</ignore>
>     <ignore>C:\WINDOWS/system32/CatRoot</ignore>
>   </syscheck>
>
>   <rootcheck>
>
> <rootkit_files>/opt/ossec/etc/shared/rootkit_files.txt</rootkit_files>
>
> <rootkit_trojans>/opt/ossec/etc/shared/rootkit_trojans.txt</rootkit_trojans>
>   </rootcheck>
>
>   <global>
>     <white_list>127.0.0.1</white_list>
>     <white_list>^localhost.localdomain$</white_list>
>     <white_list> 10.6.1.250</white_list>
>   </global>
>
>   <remote>
>     <connection>secure</connection>
>   </remote>
>
>   <alerts>
>     <log_alert_level>1</log_alert_level>
>   </alerts>
>
>   <command>
>     <name>host-deny</name>
>     <executable> host-deny.sh</executable>
>     <expect>srcip</expect>
>     <timeout_allowed>yes</timeout_allowed>
>   </command>
>
>   <command>
>     <name>firewall-drop</name>
>     <executable>firewall-drop.sh</executable>
>     <expect>srcip</expect>
>     <timeout_allowed>yes</timeout_allowed>
>   </command>
>
>   <command>
>     <name>disable-account</name>
>     <executable>disable-account.sh</executable>
>     <expect>user</expect>
>     <timeout_allowed>yes</timeout_allowed>
>   </command>
>
>   <command>
>     <name>route-null</name>
>     <executable>route-null.sh</executable>
>     <expect>srcip</expect>
>     <timeout_allowed>yes</timeout_allowed>
>   </command>
>
>
>   
>   <active-response>
>     
>     <command>host-deny</command>
>     <location>local</location>
>     <level>6</level>
>     <timeout>600</timeout>
>   </active-response>
>
>   <active-response>
>     
>     <command>firewall-drop</command>
>     <location>local</location>
>     <level>6</level>
>     <timeout>600</timeout>
>   </active-response>
>
>   
>
>   <localfile>
>     <log_format>syslog</log_format>
>     <location>/var/log/authlog</location>
>   </localfile>
>
>    <localfile>
>     <log_format>syslog</log_format>
>     <location>/var/log/syslog</location>
>   </localfile>
> </ossec_config>
>
>
> Hope you can get a clue from all this.
>
> Many thanks!
>
> Erik
>