IDSUtil

This is the companion package to the Wireshark IDS Alert plugin, it provides database maintenance and pcap file processing tools necessary for using the plugin with the Snort and Suricata IDS engines.

An overview of this package and demo of the plugin can be viewed here.

Here's the link for the pcap mentioned in the video: sansholidayhack2013.pcap.

IDSUtil provides you with the ability to have any number of IDS configurations available so you can rapidly switch between them when using the plugin in order to compare differences between IDS engines, their configuration, and/or rulesets.

Please refer to the manpages for details on the ids-util tool and ids-dbutilids-rules, and ids-pcap scripts.

The packages provided have been developed and thoroughly tested on Centrych, but they should be compatible with any Ubuntu 12.04 flavor.

If you plan to use a different distribution, or prefer a DIY approach, you can obtain the source for this package, which includes patch files for both Barnyard2 and Wireshark 1.10.6 from:


The remainder of this file is broken into two sections.

The first, immediately below, provides a brief overview for installing IDSUtil in Centrych, creating two sample configuration instances, and processing a set of sample pcap files.

The second section, further below, provides a brief discussion on what is required to update a version 107 schema for use with the Centrych modified Barnyard2 and IDSUtil.


Installation
Add Centrych Security PPA
sudo add-apt-repository ppa:centrych/security
sudo apt-get update


Database engines (either or both can be installed)
MySQL
sudo apt-get install mysql-server
NOTE: If you assign a password to the root account you will need to use the -w option with the ids-dbutil script mentioned below.

PostgreSQL
sudo apt-get install centrych-postgresql
NOTE: This package provides you with the ability to assign a password to the postgres admin user.


Web-based Database maintenance (optional)
For MySQL
sudo apt-get install phpmyadmin
NOTES:
  • Select apache2
  • Configure database: yes
  • Enter root password (if defined)
  • Application password: leave blank (generated)
For PostgreSQL
sudo apt-get install phppgadmin
NOTE: To enable login with the postgres user you will need to edit /etc/phppgadmin/config.inc.php and change the argument for the following line to 'false':

$conf['extra_login_security']

Restart apache2
sudo /etc/init.d/apache2 restart
After apache2 restarts confirm access for root and/or postgres users


Snort and Suricata
Snort
sudo apt-get install snort

Suricata
sudo apt-get install suricata
NOTE: Either package will also install python-idsutil and pulledpork packages.


Configuration Instances
While each configuration instance is contained in a separate subdirectory, it's recommended that you create a common parent directory for all instances. This allows you to collect the pcap files you're working with in one location so that each instance can be easily launched to process the pcap files.

In this example we're going to create subdirectory named 'work' in our home directory:
mkdir ~/work
cd ~/work

Once we've changed to that directory we're going to copy over some sample pcap files that were included with IDSUtil:
cp /usr/share/idsutil/pcap_samples/* .

The commands used to create the instances will be slightly different depending on if you installed one or both database engines.

If you installed both IDS engines and both databases, the following four commands should be used:

Snort
ids-dbutil -t mysql -e snort make
ids-dbutil -t mysql [-w] -d ./snort/db create
NOTE: The -w option is only needed if the MySQL admin user has a password assigned to it.  Do not include the square brackets if you need to use this option.

Suricata
ids-dbutil -t pgsql -e suricata make
ids-dbutil -t pgsql -d ./suricata/db create

If you installed both IDS engines, but only one database, the database name option is added so that switching between databases within the plugin can be accomplished by changing one character. To do that, the following four commands should be used:

Snort
ids-dbutil -t <dbtype> -e snort -n idsdb1 make
ids-dbutil -t <dbtype> [-w] -n idsdb1 -d ./snort/db create

Suricata:
ids-dbutil -t <dbtype> -e suricata -n idsdb2 make
ids-dbutil -t <dbtype> [-w] -n idsdb2 -d ./suricata/db create
NOTES: 
  • Use 'mysql' or 'pgsql' (without quotes) for the database you've installed.
  • The -w option is only required for MySQL databases that have a password assigned to the root admin account.

Rules Processing
Before you can process pcap files you will need to populate the databases with rules. Although you only have to execute a single command, there is actually a two-step process that takes place for rules processing.

The first step involves retrieving the current ruleset and extracting the necessary files into the ./<instance>/incoming directory. The ids-util tool accomplishes this first step by reading the configuration in the ./<instance>/idsutil.conf file then running the script /usr/share/idsutil/process-rules.sh in a sub-shell. This script is configured to launch the pulledpork.pl script to retrieve and process the downloaded ruleset(s).

The instance subdirectory contains the pulledpork configuration files that are used to perform the first step, which are pre-configured to download the Emerging Threats open ruleset and extract the downloaded rules into individual files based on rule category. This is done so that the rules category can be displayed in Wireshark plugin.

The next step processes each rules file in the ./<instance>/incoming directory to update the information contained in the database. After the files have been processed ids-util updates the contents of ./<instance>/sid-msg.map and ./<instance>/rules/<idstype>.rules files.

The order that these rules files are processed is defined in the file ./<instance>/default. The wrapper script calls ids-util with the STATIC_FILES then RULES_FILES as arguments. The reason there are two environment variables is because the files in the RULES_FILES variable are relocated to a subdirectory after processing so that you can more easily determine if any new categories have been added to the ruleset(s) you download.

Also, the files listed in the RULES_FILES variable contain the currently known categories for the VRT registered, ET open, and Community rulesets.  Be aware that the order of these files are different for Snort and Suricata.

Before you process rules you should review the pulledpork.conf and *sid.conf files and make any necessary changes.

Once ready, the following commands will update the rules information in each instance:

ids-rules ./snort/default --list
ids-rules ./suricata/default --list

The --list option results in the creation of a file that contains a listing of all rules that were processed. The file will be located in the database working directory, which is composed as ./<instance>/db/<dbtype>/<dbname>


Local-only rules processing
The option --local can be used if you are only interested in running the IDS engine against rules you've written.  This option will not call pulledpork.  However, to ensure that only local rules are processed, all existing rules files in the incoming directory are still deleted to ensure that only rules files defined in the STATIC_RULES variable are processed.

NOTE: The local.rules file must not be located in ./<instance>/incoming, which is cleared of all rules file when ids-rules is executed.


Pcap Processing
Once the ruleset(s) have been processed you're now ready to process the sample pcap files with the following commands:
ids-pcap ./snort/default *.pcap
ids-pcap ./suricata/default *.pcap

Once all of the files have been processed you can run the following commands to list a summary of the number of alerts that were generated for each pcap file:
grep ".pcap: generated" ./snort/db/mysql/idsdb/process_pcap.log
grep ".pcap: generated" ./suricata/db/pgsql/idsdb/process_pcap.log

The database directories shown here assume that you've installed both IDS engines and databases and used the first set of commands to create the instances.


Barnyard2 pcap processing

This section details the steps necessary to upgrade a version 107 schema. Any changes that may be required to the applications using the 107 schema is beyond the scope of this discussion.

NOTE: Please do not attempt the following on a production database without first testing the results against a copy first!

The following discussion assumes that you are using Barnyard2 with Snort and MySQL. The database is named 'idsdb' and the root account has a password assigned to it.

If your setup differs from those assumptions you will need to adjust the following accordingly.

If you're currently running Barnyard2 you will need to stop it with the following command:
sudo /etc/init.d/barnyard2 stop

The Centrych Security PPA contains a package with the modified version of Barnyard2, which can be installed via the following command:
sudo apt-get install barnyard2

The Centrych version of Barnyard2 has a configuration that can use either Snort or Suricata. Please review the /etc/default/barnyard2 file as well as the /etc/snort/barnyard2.conf or /etc/suricata/barnyard2.conf files to get a better understanding where things are located.

If you haven't already modified it, please add -p to the OPTIONS variable in /etc/default/barnyard2 for compatibility with IDSUtil.

Next, remove the current waldo file:
sudo rm /var/log/barnyard2/waldo

You should also make sure that any previously processed unified2 log files are removed so that they are not re-processed after upgrading the database.

The ids-dbutil script can be used to backup the existing v107 database with the following command:
sudo ids-dbutil -t mysql -w backup
The backup file will be written to /var/lib/idsutil/mysql/idsdb/idsdb.sql.

Upgrading the database is accomplished with the following command:
sudo ids-dbutil -t mysql -w -k upgrade

The next step is to configure the idsutil.conf file in /etc/snort. This file is used by ids-util to update the rules information in the database:
sudo ids-util --cfgfile /etc/snort/idsutil.conf --save --dbtype mysql --ssl disable
This command assumes that SSL is not used for connecting to the database.

Finally, if you're using the Centrych provided Snort package the following command can be used to update rules information in the database:
sudo update-snort-rules

If you're using Snort from a different distribution you will need to adapt the files and scripts that are used for processing rules of a configuration instance.

One final note. The Barnyard2 package was compiled with debugging enabled, but you will need to uncomment the associated environment variable in /etc/default/barnyard2 in order to have debugging output added to /var/syslog.