Manual Reference Pages  - SHOKI.TOPOLOGY (7)


shoki_sez - overview of shoki NIDS topology


     Functional Nomenclature
     The Sensor
     The Aggregator and Database
See Also


This is an overview of the basic topology of the shoki NIDS. It is intended to give the analyst an idea of how data is processed by shoki from the time it is pulled off the wire to when it shows up as an alert.

    Functional Nomenclature

There are three main groups of functions in shoki. These functions may be shared on the same machine, or may reside in different machines.
Sensor This is the machine (or machines) on which data is collected.
  A centralised server to which data are copied. This is were most of the shoki widgets live.
Database The server running Postgres. Data from all the interesting packets ends up in various database tables here.

    The Sensor

Flow of data on the sensor machine is straightforward:

sensor(8) -> raw pcap data -> lexer(1) -> pcap data

  1. Packets are captured off the wire by the sensor(8) widget and written to pcap dumpfiles
  2. Each packet dump is run through lexer(1) with a simplified set of filter rules to produce a smaller dumpfile
  3. The aggregator machine collects the reduced dumpfiles via rsync(1).

    The Aggregator and Database

Data flow through the aggregator machine and database are somewhat more complicated:

--> alerts --> shoki_sez(8) --> IM client | --> ooda(8) --> doctrine_filters | | --> lexer(1) --> events <-- | | | | -- pcap data --------> dlex(8) <---------

  1. Dumpfiles collected from the sensor machine are passed through lexer(1) running with a full set of NIDS signatures, logging matches to the events table of the shoki database.
  2. The ooda(8) widget listens to the events table, and passes new events through its doctrines (see shoki.doctrine(5) for details), potentially writing new filters to the doctrine_filters table or writing alerts to the alerts table.
  3. The dlex(8) widget listens to the doctrine_filters table, and runs any new filters against the same packet dumps read by lexer(1) in step 1 above, potentially writing new events to the events table (which will be seen by ooda(8), effectively looping back to step 2 above).
  4. The shoki_sez(8) widget listens to the alerts table, forwarding any new alerts to its contact list via the Jabber protocol.


.An Stephen P. Berry <>

More information can be found at the shoki homepage:


Check the README at the top of the source tree.



January 13, 2004 SHOKI.TOPOLOGY (7) shoki
Generated by manServer 1.07 from shoki.topology.7 using doc macros.