Server Security: OSSEC Integrates Slack and PagerDuty

We leverage OSSEC extensively to help monitor and protect our servers. If you are not familiar with OSSEC, it is an open source Intrusion Detection System (HIDS); it has a powerful correlation and analysis engine that integrates log analysis, file integrity monitoring, centralized policy enforcement, rootkit detection, real-time alerting, and active response.

It provides complete coverage if you are looking for an endpoint (server) security solution.

If you have not used OSSEC before, I recommend reading my guide to get started:
http://dcid.me/texts/my-ossec-setup-manual.html

Note that it requires root access to your servers and is meant for network and server administrators with Linux skills.

Integrating OSSEC with Slack and PagerDuty

One of the new features that we open sourced and pushed to OSSEC is an “integrator” daemon that we have been using internally to connect OSSEC to external APIs and alerting tools. The first two APIs we officially added were for Slack and PagerDuty.

To get started, you have to download OSSEC from our fork, either from the development branch or from the our stable branch:

After you install (or update) OSSEC, you need to enable the integrator daemon with the following command:

# /var/ossec/bin/ossec-control enable integrator

If the right version of OSSEC is installed, no errors will be generated and you are ready to configure it.

Integrating with Slack

Slack is taking the world by storm and many companies are using it for internal communication. Most people are always connected to it, making it more practical to use for real-time alerts than to depend solely on email.

Additionally, Slack integration is very easy to configure. First, you have to go to Slack and enable a new integration under Settings > Custom Integrations > Incoming Webhooks.

You then choose a channel and a new hook will be created in the format of:
https://hooks.slack.com/services/RANDOMSTRING/RANDOMSTRING.

Save that URL and add it in with the following to your ossec.conf file:

<integration>
<name>slack</name>
<level>10</level>
<hook_url>https://hooks.slack.com/services/RANDOM/RANDOM</hook_url>
</integration>

Once you restart OSSEC, it will force all alerts with level above 10 to go to the OSSEC room you chose before, just like this:

You can also specify per alert category or rule id, by using the <group> or <rule_id> entries. So if you only want to receive all authentication related alerts via slack, you can add the following to your config:

<integration>
<name>slack</name>
<group>authentication</group>
<hook_url>https://hooks.slack.com/services/RANDOM/RANDOM</hook_url>
</integration>

Restart OSSEC and you are done.

Integrating with PagerDuty

PagerDuty is one of the best products out there to manage and notify network and security administrators (DevOps/SecOps) about issues that deserve immediate attention.

It is a powerful tool and one that can now be integrated with OSSEC. Be careful not to overdo it with this one. Only push to PagerDuty events that really matter.

For example, we have some rules that are categorized as “immediate_response_required” as they tend to be related to service availability; either a service went down or the system is running out of memory. An example would be the Nginx rule 31321 and the Syslog rule 1007:

<rule id="31321" level="5" ignore="60">
<if_sid>31303</if_sid>
<match>(99: Cannot assign requested address</match>
<description>Nginx connection error - Out of sockets available.</description>
<group>immediate_response_required,service_availability,</group>
</rule>

<rule id="1007" level="7">
<match>file system full|No space left on device</match>
<description>File system full.</description>
<group>immediate_response_requird,low_diskspace,</group>
</rule>

As you can see, this triggers an alert if Nginx is unable to assign an address (likely down) or if a log file is generating an warning about no space left. These are generally pretty serious as they affect the availability of the server. So they can be a good one to push straight to Pagerduty for immediate attention.

To do that, you first need to get your API key from the Pagerduty site. Go to Configuration > Services and view the service you have your escalations integrated with. There, you will see the API key to use, like this:

Integration Type Generic API View PagerDuty API documentation
Integration Key RANDONKEY

Save the integration key and copy to the ossec.conf:

<integration>
<name>pagerduty</name>
<group>immediate_response</group>
<api_key>RANDOMKEY</api_key>
</integration>

In this example we tied OSSEC to the immediate_response group, so any serious availability issue will be pushed straight to the Pagerduty escalation. Similar to the Slack integration, you can tie it to alert levels (using <level>) or to specific alert ids (using <rule_id>).

Adding More Integrations

That’s it for now, however, adding new integrations to OSSEC is very easy and we plan to add more very soon. Hipchat is the next one on our list.

Are there any other integrations you want to see? Let us know and we will make it happen.