It was sunny and pleasant throughout the two days of OSSEC CON in Cupertino, California. This is the first of its kind and we did not know what to expect. The agenda says 11:00am to 1:00pm Thursday for registration and lunch and I was thinking people probably won't show up until 12 or 12:30pm. When the first attendee appeared at 11 AM sharp, we were pleasantly surprised. Gradually, more people arrived and we had boxed lunch and casual conversation throughout lunch period. The atmosphere was that people were simply happy to meet other long-time OSSEC users face-to-face, as if you would expect in a 20 year high school reunion.
By the way, you can get the full agenda and introductory slides for the symposium here.
OSSEC CON Day 1
The OSSEC project, as most of you already know, was founded by Daniel Cid almost 10 years ago. In 2008, Third Brigade acquired the OSSEC project, and in 2009, Trend Micro acquired Third Brigade. Fast forward to 2012: the time has come for Trend Micro to reach out to the OSSEC user community, actively seeking feedback and contributions. The OSSEC vision for 2012 is "Three in One", as shown in the following diagram.
Trend Micro's strategy for OSSEC can be summarized as:
- Promote Trend Micro brand through OSSEC Project sponsorship
- Engage in Open Source development and learn from the community
- Ensure timely technical support for Trend Micro’s paid support customers
The community, which consists of OSSEC Contributors and Users, have these goals in general:
- Provide the OSSEC user community with regular bug fixes and new features.
- Provide timely announcements of OSSEC rule and platform enhancements thru the new OSSEC website and social media
- Promote OSSEC and Trend Micro brand and development through regular community meetings
- Increase adoption of the OSSEC HIDS platform.
Going forward, we expect these three entities to collaborate closely and benefit each other. The OSSEC project will remain open and free, while together we will set the direction for future enhancements and work on future releases.
After a brief introduction of all attendees, Trend Micro Legal Counsel, John Chen, appeared on stage and restated that the OSSEC source code and rules are still governed under GPL license terms, as it has been stated on the OSSEC License page.
The center piece of presentation on Day 1 was Keynote I – Experiences Applying OSSEC. Michael Starks, in his Ninja outfit (just kidding), energized the audience with his humor and extensive wisdom that one can only expect from a very experienced user of OSSEC. His presentation slides can be found here.
The keynote led to an open discussion by all participants about their pain points, which are summarized into several categories at the end of this article.
OSSEC CON Day 2
Day 2 started with continental breakfast (9:00am ~ 9:45am) and again we were pleasantly surprised to see the first attendee arrived before 9am. The enthusiasm was amazing throughout the second day as well. First, Trend Micro presented its product strategy; in particular, the DeepSecurity product lineup, in a world fast moving to the virtualized cloud environment. While OSSEC is a very capable tool for HIDS (Host-based Intrusion Detection System), DeepSecurity is a powerful and versatile HIPS (Host-based Intrusion Prevention System) that meets enterprise server security needs, including Anti-Virus, Web Reputation, Firewall, in addition to the standard OSSEC features of File Integrity and Log Inspection. Interested users are advised to contact Trend Micro for more details.
Next, experiences with high volume OSSEC deployment were shared. One installation had 3,000 OSSEC Windows agents and 500 Unix agents (mix of HP / AIX / Linux / SunOS). This was before 'authd' was available and the deployment consisted of three steps:
- Use OSSEC make option to set maximum number of agents to be greater than 256 (default). Generate all 3500 keys in advance, leaving the IP addresses as 'ANY' in OSSEC manager's clients.keys file initially.
- After basic install and as the agents came up and checked in with the manager, the real agent IP addresses were placed into clients.keys file by scripts.
- Each agent waited for extra 30 seconds to allow the manager to push agent.conf to the agent and merge the configuration together, then restart the agent.
Some manager performance tuning was performed. For example, the Unix fs cache was set to 10%, from the default value of 90%. This allowed OSSEC 'analysisd'/'remoted' to grab as much memory as needed. As for server spec, one manager was able to handle 4,000 agents with 72 GB RAM and 1 TB disk. The installation had 'logall' turned on to MySQL database, while alerts are piped to Splunk for noise reduction and further correlation. The 500 MB limitation on free version of Splunk was not a problem at all in this example.
Next, a list of 3rd party add-on tools were discussed. Splunk topped the list as the most popular among attending participants. Others include OSSIM, logstash, logZilla, ELSA, ArcSight, EnVision, LogCentral, AnaLogi. A demonstration of OSSEC and ELSA was shown and interested users can refer to http://code.google.com/p/enterprise-log-search-and-archive/ to find more about ELSA.
After a nice lunch break, Day 2 afternoon started with Keynote II – OSSEC Feature Wish List. Michael Starks took the stage again with his charm. He first used an example to illustrate the importance of having a shared vision; the lack of a shared vision and realistic goals often puts a project on hold. Michael painted the future of OSSEC with the following wish list:
- Interoperability based on standards, e.g., common standard for rule writing. This should result in better OSSEC rules taxonomy and ease of understanding
- Interface well with other tools such as CVE, OVAL, syslog-ng, etc., adapting format such as CEE (Common Event Expression http://cee.mitre.org/)
- A collective attitude which encourages and supports contributions
It was noted that to be an OSSEC contributor, writing code is not necessary. Everyone has something to offer. Can you submit a bug report? Are you an artist? Do you have an idea? With a friendly and encouraging community, talent naturally emerges. The presentation slides can be found here.
The Big OSSEC TO DO List
To summarize the pain points as well as the wish list, the complete list is shown below:
Dealing with BugsReducing the noise level – Tune rules to avoid alerts flooding
- JB Cheng is coordinating the next OSSEC minor release, which will focus on bug fixes.
- Michael is helping to prioritize existing bugs
- Existing patches contributed by users will be collected and included after passing review.
- Roadmap will be communicated and beta testers of the new release will be recruited.
- Suggestion to document the current development process in an FAQ section.
- Ideally an auto regression framework can be built to make sure bug fixes do not break existing functionality. This is to be considered.
Reducing the noise level – Tune rules to avoid alerts flooding
- In general, Linux rules are OK; but Windows rules generate too many alerts by default.
- Suggestion to gather feedback and adjust Windows rules in future releases. By default, email alerts should be meaningful and actionable (stay consistent and informative) while the rest can be viewed in a GUI.
- One use case is piping OSSEC alerts to Splunk and use Splunk rules to consolidate chatty alerts.
- One attendee stated that the <do_not_group> configuration option for putting multiple alerts in one email does not seem to work as expected (to be verified).
- Suggestion to tie rule alert levels with business risk factor for different assets, so non-critical assets do not trigger email notifications.
- Suggestions to tie alert levels with different group characteristics. For example, admin group vs. normal groups.
Making agent deployment easier – Key exchange
- Before 'authd' was available, deploying lots of OSSEC agents was very time consuming.
- 'authd' works on Linux, but some users experienced problems on Windows although there were successful cases too. Suggest documenting the success stories of 'authd'.
- Still, agent key management is a barrier to entry
- Making Windows Agent MSI build can allow the leverage of Active Directory group policy for Windows agent deployment
- RedHat RPMs, Debs, etc. are good too.
- Example: some experience showed Puppet was good for Unix systems.
- Example: OCS-ng was also used in at least one deployment.
- It should "just work".
Decoupling Rule and Decoder Updates
- Current OSSEC rules are coupled with OSSEC releases. By decoupling them it will allow faster response to current and emerging threats with faster rule fixes.
WUI – Web User Interface
- WUI 0.3 has some incompatibilities with OSSEC 2.6, which makes it hard to use.
- IP address and Date field got mixed up.
- WUI reporting can be enhanced to become more useful.
- Viewing alerts is currently via text display only. It would be nice to have CSV output option.
Web site – www.ossec.net
- The web site has been restructured with a new look and better navigation. Please report any broken links or missing information.
- The Wiki feature on the old web site has been discontinued due to extensive spam trouble. Important content will be reproduced as static pages at user's request.
Suggestions to provide/link to tutorial type of information such as:
- A live demo OSSEC server, ideally with agents in virtual machines
- Tutorial video on YouTube (any volunteers?)
- Re-visit the Virtual Machine image effort which may contain OSSEC manager , WUI and even some 3rd party tools preinstalled.
Interface with 3rd party tools
- e.g., Snort –> OSSEC –> Splunk
- e.g., OSSEC –> ELSA
- e.g., Hadoop log –> OSSEC (feasible?)
- Suggestion on syslog output enhancement – adding File Integrity checksum to alert output so it can be used for lookups (refer to : http://www.ossec.net/doc/manual/rules-decoders/rule-lists.html and http://cr.yp.to/cdb.html)
That's it folks! The first OSSEC Symposium is successfully concluded! Now, let’s get busy!
JB Cheng – 7/16/2012
Michael Starks – 7/17/2012
Vic Hargrave – 7/17/2012