LibpreludeDB and IDMEF wrapper functions

What is it?

LibpreludeDB is a new database handling layer for Prelude IDS. The project aims at creating a shared library allowing application to store and retrieve IDMEF alerts from databases in various formats and various storage types (SQL servers, files, etc.). The first version has been done as an end-of-semester project for my SQL class, however later the code has been totally rewritten.

See description, structure diagram (in dia), IDMEF draft v.0.6 (which currently Prelude is based on), IDMEF draft v.0.10 (latest).

This work is integrated with implementation of IDMEF wrapper functions in libprelude. For performance reasons, Prelude does internal processing of IDMEF data using its own data structures, rather then XML. The purpose of IDMEF wrapper functions is to create a set of functions allowing programmer to access Prelude's IDMEF data structures, abstracting these structures themselves, thus allowing future changes to IDMEF implementation.

Code status:

The code is under development in separate CVS branch. This branch has been forked off HEAD soon after release of 0.8.x version, so it should be mostly compatible. However all sensors have to be recompiled to in order to work with modified libprelude correctly. This is an experimental work. Use at your own risk. You have been warned.

Currently the following capabilities are provided:

[libprelude] Implementation of IDMEF wrapper functions for each object class
[libprelude] Abstracted interface to any value on the IDMEF tree by idmef_object_t and idmef_value_t objects. For an usage description see my mail (edited) to Laurent describing a proof-of-concept debug2 plugin in Prelude Manager. Also see this announcement (edited) for further description.
[libprelude] Matching of in-memory alerts against a specified criteria set.
[libpreludeDB] IDMEF writing to MySQL and PostgreSQL databases, in a format used by Prelude 0.8.
[libpreludeDB] IDMEF reading from MySQL and PostgreSQL databases, in a format used by Prelude 0.8 (partally implemented). First announcement.
[Prelude-Manager] Generic database reporting plugin db to output alerts using libpreludeDB.
[Prelude-Manager] debug2 reporting plugin for testing / playing with new IDMEF interface.

Code availability:

The code can be checked out from db-work branch of Prelude CVS using following commands:

Libprelude with IDMEF wrappers:
cvs -d:pserver:anonymous@cvs.prelude-ids.org:/cvsroot/prelude co -r db-work libprelude
You can browse it via CVSWeb here.

LibpreludeDB:
cvs -d:pserver:anonymous@cvs.prelude-ids.org:/cvsroot/prelude co libpreludedb
You can browse it via CVSWeb here.

Prelude Manager modified to use libpreludeDB and IDMEF wrappers:
cvs -d:pserver:anonymous@cvs.prelude-ids.org:/cvsroot/prelude co -r db-work prelude-manager
You can browse it via CVSWeb here. For using the libpreludeDB-based db output plugin it may be a good idea to review the description of DB interface parameters in libpreludeDB's README.

Other useful code:

List of all leaf nodes in IDMEF tree. The paths are relative to top-level idmef_message_t object; the notation is compatible with the one used by idmef_object_new(). This list has been created using a quick and dirty generator.
idmef-send.c, a tool for sending alerts from command line (using abstracted IDMEF interface). Description, sample input file and corresponding output from Prelude Manager.
idmef-test.c, a simple tool used for testing IDMEF message extraction. Description.
idmef-demo.c, a simplified (and more readable) version of idmef-test.c. Description for idmef-test.c is also applicable here. [Note: the file has not been updated to reflect API change introduced on 2003-05-05, so will not work for now.]
sql-test.c, a simple tool for performing SQL queries, usually used for testing. Description. For detailed explanation of the DB interface specification, see libpreludeDB's README. Note that since you are not using format plugins, the format parameter is not required.
patch for skeleton filter plugin in Prelude Manager, which allows filtering based on substring matching in alert field. For example, invoking Prelude-manager with:
prelude-manager --skeleton -h textmod -r alert.analyzer.model/NIDS -r alert.analyzer.manufacturer/Prelude --textmod -s
will pass to textmod plugin only alert messages where alert.analyzer.model field contains substring NIDS and alert.analyzer.manufacturer field contains substring Prelude.
This should be viewed as proof-of-concept only code for abstracted IDMEF interface.
[Note: the file has not been updated to reflect API change introduced on 2003-06-01, so will not work for now.]

Recent important changes:

(detailed CVS change log can be found here, here and here ):

[2003-06-01] Removed idmef_cache_t, using Nicolas' hash-based cache within idmef_message_t instead. The API should be cleaner now.
[2003-05-19] Partial reimplementation and API change of database IDMEF extraction. New version of idmef-test.c (Nicolas).
[2003-05-05] idmef_string_t reimplemented (me). Compatibility layer for Prelude 0.8 sensors implemented atop of new IDMEF layer, original IDMEF implemention removed (me). A lot of changes to libpreludeDB alert reading and a new version of idmef-test.c (Nicolas). idmef-demo.c not yet updated.
[2003-02-28] IDMEF enumeration support. Added idmef-send.c.
[2003-02-15] Partial IDMEF reading support (finally!). Added idmef-test.c, idmef-demo.c
[2003-01-04] Huge rewrite of SQL layer done by Nicolas Delon. New version of sql-test.c, corresponding to API changes and including some more tests.
[2003-01-02] Created this page :)

Back to my home page.