Saltstack – OSSEC state using reactor

After concentrating on learning Salt over the last few weeks I have been constantly surprised by how simple yet powerful the system is. When I first began, I started by setting up my base configurations (users, iptables, ssh, etc) into states and as I progressed things began getting more and more sophisticated.

For the most part everything has been completely smooth but as with learning anything new I have gotten stuck from time to time trying to figure out how to do something or troubleshooting a formatting error I introduced. Luckily with such a great and active community as well as great documentation these rough spots have been few and far between.

With this state I hope to go over some of these rough spots that I ran into, in the hope that it will help other people. The following state will:

• Transfer the OSSEC software to a minion
• Compile and install OSSEC using defined settings from pillar
  • Thanks to daviddyball‘s example which really helped me get started
• Start up the ossec-authd service on the master using Saltstack reactor / event system
• Start up the ossec-authd service on the agent
• Negotiate keys between the master and agent
• Shut down the ossec-authd service on the master using Saltstack reactor / event system
• Start the OSSEC services

For this setup the config will look like this (along with a change to /etc/salt/master):

/srv/reactor
  - ossec.sls
/srv/pillar
  - top.sls 
  - servers.sls
  - packages.sls
/srv/pillar/service-ossec
  - init.sls
/srv/pillar/service-openssl
  - init.sls
/srv/salt
  - top.sls
/srv/salt/linux-server/ossec
  - init.sls
  - agent.sls
  - server.sls
  - preloaded-vars.jinja
  - local_rules.xml
/srv/salt/linux-server/ossec/files
  - ossec-hids-2.7.tar.gz
/srv/salt/linux-server/build-utils
  - init.sls
  - ssldev.sls

Reactor / Event System

To begin breaking this down I figured I would start with one aspect that took me a little while to get a grasp on, reactor and the event system.

In order to negotiate keys automatically, OSSEC has a service called ossec-authd that can be started from the server and the client to automatically generate and distribute keys. The only problem is it will generate/distribute a key for any ossec-authd request so the OSSEC manual recommends you do not leave it running all the time.

After some research I figured this would be a good way to try out the reactor and event system. So, to begin using reactor I added the following to my /etc/salt/master file:

reactor:
  - 'ossec':
    - /srv/reactor/ossec.sls

The above configuration will listen for any event that is tagged as “ossec” and will “react” by running the commands in the ossec.sls file which can be configured in much the same way as a state.

All events will have a tag and data associated with it (as well as other things such as the minion name). According to the Saltstack documentation “A “tag” allows for events to be filtered.” which makes parsing large amounts of event data more efficient. In order to see these events and the data they contain Saltstack has provided a python script that can be run to watch for events. This was invaluable to me when setting up my event:

import salt.utils.event
event = salt.utils.event.MasterEvent('/var/run/salt/master')
for data in event.iter_events(tag='auth'):
    print(data)

We now have reactor started and listening for events tagged as ossec but we need to setup our ossec.sls file in order to start doing something with these events. The following .sls file will look at the data of the event and perform different commands on our ossec server based on the data. Ultimately I would like to define the tgt and the path using pillar but as of this writing I cant get it working with pillar.

{% if data['data'] == 'ossec-auth-start' %}
ossec-auth-start:
  cmd.cmd.run:
    - tgt: ubuntu01
    - arg:
      - /var/ossec/bin/ossec-authd -p 1515 >/dev/null 2>&1 &
{% elif data['data'] == 'ossec-auth-stop' %}
ossec-auth-stop:
  cmd.cmd.run:
    - tgt: ubuntu01
    - arg:
      - pkill ossec-authd >/dev/null 2>&1
{% endif %}

I am getting ahead of myself but in order to have a complete example I thought it would be useful to see. Later, in the state the client will execute the following salt-call in order to fire an event with the ossec tag and trigger the reactor:

salt-call event.fire_master 'ossec-auth-start' 'ossec'

Pillar

For my configuration I am trying to design as much as possible around pillar. I have centralized configuration files for my servers and desktops where I can define anything that is system dependent (IP address, what services are running, etc) and then I have service-<SERVICENAME> pillars to define anything that is application specific (paths, ports, uid/gid, etc).

To begin I will define my service-ossec pillar by creating a new directory called /srv/pillar/service-ossec and creating an init.sls inside of the directory with the following values:

service-ossec:
  version: '2.7' {# version of software to be installed (taken off of .tar file) #}
  language: 'en'
  userdir: '/var/ossec' {# Installation path #}
  en_active_response: 'y' {# Enable active response #}
  en_syscheck: 'y' {# Enable system checks #}
  en_rootcheck: 'y' {# Enable rootkit detection #}
  en_update_rules: 'y' {# Update rules #}
  en_syslog: 'y' {# Enable syslog checks #}
  server_ip: '172.16.0.10' {# IP of the OSSEC master server #}
  en_email: 'y' {# Enable email notifications #}
  email_address: '[email protected]' {# Email address to send notifications #}
  white-list: '172.16.0.0/16' {# IP whitelist #}

Since I also use openssl for a variety of services in my environment I have a centralized service-openssl pillar defined in /srv/pillar/service-openssl/init.sls. These values are then used and shared between multiple programs in my environment. One of which is OSSEC.

service-openssl:
  defaultbits: 2048
  countrynamedefault: US
  stateorprovincenamedefault: California
  orgnamedefault: My Company Name

Next, I use a centralized packages.sls pillar so I can deal with differing package names between Debian and RedHat based systems. While this list appears to be growing, here is what I have now:

packages:
{% if grains['os_family'] == 'Debian' %}
  apache: apache2
  php: libapache2-mod-php5
  git: git-core
  snmp: snmpd
  snmp-service: snmpd
  ssh: ssh
  ssh-service: ssh
  vi: vim
  ntp: ntp
  ntp-service: ntp
  python-mysql: python-mysqldb
  mysql-service: mysql
  winbind: winbind
  samba-service: smbd
  build-env: build-essential
  ssl-dev: libssl-dev
  auth-pam: libapache2-mod-auth-pam
{% elif grains['os_family'] == 'RedHat' %}
  apache: httpd
  php: php
  git: git
  snmp: net-snmp
  snmp-service: snmpd
  ssh: openssh
  ssh-service: sshd
  vi: vim-enhanced
  ntp: ntp
  ntp-service: ntpd
  python-mysql: MySQL-python
  mysql-service: mysqld
  winbind: samba-winbind
  samba-service: smb
  build-env: make
  ssl-dev: openssl-devel
  auth-pam: mod_auth_pam
{% endif %}

Now, I will update my existing servers.sls and define a server to be the OSSEC server and one to be the OSSEC agent. Please note that I have included more information in this example to help detail how I am using this .sls file but for the purposes of this example all I really need to define here is the roles ossec.agent and ossec.server:

servers:
  ubuntu01:
    roles:
      - network
      - ssh
      - ossec.server
      - users
      - iptables
      - apache  
      - owncloud
    websites:
      - owncloud
    tcp_ports:
      - 80
      - 443
    domain: domain.local
    gateway: 172.16.0.1
    gwdev: eth0
    dns:
      - 172.16.0.10
      - 172.16.0.11
    network_adapters:
      eth0: {
      en: 'True',
      ip: 172.16.0.10,
      sn: 255.255.0.0,
      nw: 172.16.0.0,
      bc: 172.16.255.255 }
      eth0:1: {
      en: 'True',
      ip: 172.16.0.100,
      sn: 255.255.0.0, 
      nw: 172.16.0.0, 
      bc: 172.16.255.255 }

  centos01:
    roles:
      - network
      - users
      - iptables
      - ossec.agent
    tcp_ports:
      - 80
      - 443
    domain: domain.local
    gateway: 172.16.0.1
    gwdev: eth0
    dns:
      - 172.16.0.10
      - 172.16.0.11
    network_adapters:
      eth0: {
      en: 'True',
      ip: 172.16.0.20,
      sn: 255.255.0.0, 
      nw: 172.16.0.0, 
      bc: 172.16.255.255 }

Lastly, now that both pillars are setup we need to make sure they are defined in the /srv/pillar/top.sls:

base:
  '*':
    - servers
    - packages
    - service-ossec
    - service-openssl

State – top.sls

While we are still close to the topic of how I setup my pillar’s I wanted to touch on how I setup my state top.sls file. My state top.sls file will loop through the roles defined in the servers.sls and apply the corresponding state based on its OS type using grains. So my /srv/salt/top.sls file looks like this:

{% set hostname=grains['id'] %}
{% if grains['kernel'] == 'Linux' %}
{% set kernel='linux' %}
{% elif grains['kernel'] == 'SunOS' %}
{% set kernel='solaris' %}
{% elif grains['kernel'] == 'Windows' %}
{% set kernel='windows' %}
{% elif grains['kernel'] == 'Darwin' %}
{% set kernel='macintosh' %}
{% endif %}

base:
  '*':
    - common.salt

  'os_family:Debian':
    - match: grain
    - common.apt

  {% if pillar['servers'][hostname] is defined %}
  {% for roles in pillar['servers'][hostname]['roles'] %}
  'servers:{{ hostname }}:roles:{{ roles }}':
    - match: pillar
    - {{ kernel }}-server.{{ roles }}
  {% endfor %}
  {% endif %}

  {% if pillar['desktops'][hostname] is defined %}
  {% for roles in pillar['desktops'][hostname]['roles'] %}
  'desktops:{{ hostname }}:roles:{{ roles }}':
    - match: pillar
    - {{ kernel }}-desktop.{{ roles }}
  {% endfor %}
  {% endif %}

When the top.sls file parses through my servers.sls file it will find the role ossec.server and ossec.agent and apply those states to the servers ubuntu01 and centos01. Since both of these servers are linux servers it will apply the ossec state from the linux-server directory.

State – linux-server.ossec

In order to create my OSSEC state I have created the following directories to hold my state files /srv/salt/linux-server/ossec and /srv/salt/linux-server/ossec/files. My state begins with the init.sls file which will:

• Install compilation prerequisites from two other states (defined below)
• Copy the ossec tarball to the server from the /srv/salt/linux-server/ossec/files directory
• Decompress the tarball
• Copy the answers to the installer prompts to the ossec installation directory
• Run the installer

In order to manage the state centrally I have defined all of the options in the service-ossec pillar we created earlier.

{% set hostname = grains['id'] %}
{% set version = pillar['service-ossec']['version'] %}
{% set ossecdir = 'ossec-hids-{0}'.format(version) %}

include:
  - linux-server.build-utils
  - linux-server.build-utils.ssldev

ossec-install-directory:
  file.directory:
    - name: /usr/src/ossec-install
    - order: 500

ossec-download-installer:
  file.managed:
    - source: salt://linux-server/ossec/files/{{ ossecdir }}.tar.gz
    - name: /usr/src/ossec-install/{{ ossecdir }}.tar.gz
    - order: 501
    - require:
      - file: ossec-install-directory

ossec-extract-installer:
  cmd.run:
    - name: 'tar -zxf {{ ossecdir }}.tar.gz'
    - cwd: '/usr/src/ossec-install/'
    - unless: stat {{ pillar['service-ossec']['userdir'] }}/bin/ossec-control
    - order: 502
    - watch:
      - file: ossec-download-installer

ossec-installer-variables:
  file.managed:
    - name: '/usr/src/ossec-install/{{ ossecdir }}/etc/preloaded-vars.conf'
    - source: 'salt://linux-server/ossec/preloaded-vars.jinja'
    - template: jinja
    - order: 504
    - watch:
      - cmd: ossec-extract-installer

ossec-install:
  cmd.run:
    - name: '/usr/src/ossec-install/{{ ossecdir }}/install.sh'
    - unless: stat {{ pillar['service-ossec']['userdir'] }}/bin/ossec-control
    - require:
      - pkg: make-package
      - pkg: gcc-package
      - pkg: ssldev-package
    - order: 505
    - watch:
      - file: ossec-installer-variables

State – linux-server.ossec.server

Next, I want to be able to manage both by OSSEC server and agent using the same configuration so I created a server.sls file which will perform a server install.

The sls:

• Manages my OSSEC local_rules.xml file
• Generates an SSL key and certificate which is used by ossec-authd
• Starts the OSSEC service
• Creates a cron job to ensure that ossec-authd is not running for very long in the case it is not shut down properly.

{% set hostname=grains['id'] %}
{% set domain=pillar['servers'][hostname]['domain'] %}

include:
  - linux-server.ossec

rules-config:
  file.managed:
    - name: {{ pillar['service-ossec']['userdir'] }}/rules/local_rules.xml
    - user: root
    - group: ossec
    - mode: 550
    - order: 510
    - source: salt://linux-server/ossec/local_rules.xml

ssl-key:
  cmd.run:
    - name: openssl genrsa -out {{ pillar['service-ossec']['userdir'] }}/etc/sslmanager.key 2048
    - unless: stat {{ pillar['service-ossec']['userdir'] }}/etc/sslmanager.key
    - order: 510

ssl-cert:
  cmd.run:
    - name: openssl req -subj
            '/CN={{ hostname }}.{{ domain -}}
            /C={{ pillar['service-openssl']['countrynamedefault'] -}}
            /ST={{ pillar['service-openssl']['stateorprovincenamedefault'] -}}
            /O={{ pillar['service-openssl']['orgnamedefault'] }}'
            -new -x509 -key {{ pillar['service-ossec']['userdir'] }}/etc/sslmanager.key
            -out {{ pillar['service-ossec']['userdir'] }}/etc/sslmanager.cert
            -days 730
    - unless: stat {{ pillar['service-ossec']['userdir'] }}/etc/sslmanager.cert
    - onlyif: stat {{ pillar['service-ossec']['userdir'] }}/etc/sslmanager.key
    - order: 510

ossec-service:
  service.running:
    - name: ossec
    - enable: True
    - sig: ossec-syscheckd
    - order: 511
    - require:
      - cmd.run: ossec-install

{# Since OSSEC does not recommend keeping authd running my workaround so far is to run a cron job
   to shut it off after a period of time. I need to figure out a better way of doing this in the
   future as this may require more than one highstate to add keys for a client and it is cheesy #}

pkill ossec-authd >/dev/null 2>&1:
  cron.present:
    - user: root
    - minute: '*/5'

State – linux-server.ossec.agent

As with the server example above, the following config sets up our agents. This part of the configuration is where we start using the event system and reactor.

The agent.sls:

• Fires an event to the salt master from the minion which will signal reactor to start the ossec-authd service on the OSSEC server
• The OSSEC agent then starts the ossec-authd service and negotiates keys with the OSSEC server
• After the negotiation is done the minion fires a new event to the salt master which will signal reactor to shut down the ossec-authd service on the OSSEC server.
• Finally the OSSEC services will be started on the agent.

include:
  - linux-server.ossec

{# Use events/reactor system to start up the ossec-authd process on the OSSEC master #}
server-auth:
  cmd.run:
    - name: salt-call event.fire_master 'ossec-auth-start' 'ossec'
    - unless: stat {{ pillar['service-ossec']['userdir'] }}/etc/client.keys
    - order: 510

{# OSSEC authd agent connects to master and registers its key #}
agent-auth:
  cmd.wait:
    - name: sleep 1 && {{ pillar['service-ossec']['userdir'] }}/bin/agent-auth -m {{ pillar['service-ossec']['server_ip'] }} -p 1515
    - unless: stat {{ pillar['service-ossec']['userdir'] }}/etc/client.keys
    - order: 511
    - watch:
      - cmd.run: server-auth

{# We are done creating our key so lets shut down the ossec-auth process on the master using reactor #}
server-auth-shutdown:
  cmd.wait:
    - name: salt-call event.fire_master 'ossec-auth-stop' 'ossec'
    - order: 512
    - watch:
      - cmd.wait: agent-auth

{# Start the OSSEC services on the agent #}
ossec-service:
  service.running:
    - name: ossec
    - enable: True
    - sig: ossec-syscheckd
    - order: 513
    - require:
      - cmd.run: agent-auth

Config File – preloaded-vars.jinja

The preloaded-vars.jinja is used by the OSSEC installation to automatically answer any questions during the install. For our configuration everything is defined in the service-ossec pillar and the options that are required for a server only show up if you have defined the ossec.server role in the servers.sls.

{%- set hostname=grains['id'] %}
USER_LANGUAGE="{{ pillar['service-ossec']['language'] }}"
USER_NO_STOP="y"
{%- if 'ossec.server' in pillar['servers'][hostname]['roles'] %}
USER_INSTALL_TYPE="server"
USER_EMAIL_SMTP="{{ pillar['resources']['smtp'] }}"
{%- else %}
USER_INSTALL_TYPE="agent"
USER_AGENT_SERVER_IP="{{ pillar['service-ossec']['server_ip'] }}"
{%- endif %}
USER_DIR="{{ pillar['service-ossec']['userdir'] }}"
USER_DELETE_DIR="y"
USER_ENABLE_ACTIVE_RESPONSE="{{ pillar['service-ossec']['en_active_response'] }}"
USER_ENABLE_SYSCHECK="{{ pillar['service-ossec']['en_syscheck'] }}"
USER_ENABLE_ROOTCHECK="{{ pillar['service-ossec']['en_rootcheck'] }}"
USER_UPDATE_RULES="{{ pillar['service-ossec']['en_update_rules'] }}"
USER_ENABLE_EMAIL="{{ pillar['service-ossec']['en_email'] }}"
USER_EMAIL_ADDRESS="{{ pillar['service-ossec']['email_address'] }}"
USER_ENABLE_SYSLOG="{{ pillar['service-ossec']['en_syslog'] }}"
USER_WHITE_LIST="{{ pillar['service-ossec']['white-list'] }}"

local_rules.xml and files/ossec-hids-2.7.tar.gz

From your OSSEC installation make a copy of the local_rules.xml file and put it in the /srv/salt/linux-server/ossec state directory so that you can manage the default OSSEC rules centrally.

At this point you will also need to download a copy of OSSEC and place it in the files directory to enable the OSSEC software installation. Please note the version number on this file as it must correspond with the version set in our service-ossec pillar.

Final dependencies

In order for OSSEC to properly install we need a compiler and the ssl-dev packages so that we can build the ossec-authd service. I used to define these in my OSSEC state but as I now have other installers that require the same thing I have moved these out to a state of their own under /srv/salt/linux-server/build-utils and I call them using includes.

Since this is pretty basic I am just going to include the contents of both files

init.sls:

make-package:
  pkg:
    - name: make
    - order: 50
    - installed

gcc-package:
  pkg:
    - name: gcc
    - order: 50
    - installed

ssldev.sls:

ssldev-package:
  pkg:
    - name: {{ pillar['packages']['ssl-dev'] }}
    - order: 50
    - installed

Final notes

I am working on putting together a full pillar / state environment to upload on to github to allow anyone to download a copy of my environment but please keep in mind the beauty of Saltstack is that it can be configured and reorganized in a variety of ways.

While this environment was built with the idea of centralized configuration using pillar I have also thought about changing things around to be a bit more dynamic. I guess that is part of the beauty of this software. It is extremely flexible.