Shoki Packet Hustler Manual

Table of Contents


The shoki packet hustler (hustler(1)) is a tool for visualisation of network traffic. Without modification it supports plotting and clustering of IP traffic. Packet variables which can be tracked by default include all standard IP, TCP, UDP, and ICMP values (i.e., anything that can be found in struct ip, struct tcphdr, struct udphdr, or struct icmp).

Hustler(1) was originally developed as a diagnostic tool for use with cluster analysis code in the shoki NIDS. In its current form, it allows an analyst to view packet variables in a number of ways: a set of related 2d plots (x-y, x-z, and y-z) of three user-specified variables; a 3d plot of the same three variables; and a 3d phase space plot of any specified variable.

In addition, filtering and clustering mechanisms allow the analyst to view (and manipulate views) of the packet data. Filtering is done via standard shoki filter rules (which can include libpcap-style filter expressions, POSIX extended regular expressions, and other features). Clustering is done via an agglomerative heirarchical clustering method (radix clustering). Details of this method can be found elsewhere in this manual.

1 Tutorial

1.1 Starting hustler(1)

In order to follow the examples given in this tutorial, you should first cd into the top of the shoki source tree. Unless otherwise noted, relative paths used in this tutorial will be relative to the top of the shoki source directory. So if the source is in /usr/local/src/shoki/shoki-0.3.0, then ./test/test_dump.gz would be equivalent to /usr/local/src/shoki/shoki-0.3.0/test/test_dump.gz.

Assuming shoki has been properly installed, hustler(1) may be invoked with no arguments. By default, it will be installed as /usr/local/shoki/bin/hustler, so a sample command line invokation would be:

	# /usr/local/shoki/bin/hustler

This should bring up a file selection dialog. If you ran hustler from the top of the shoki source tree, it will look like this:


Scroll down to the bottom of the Folders pane (the one on the left) to find the test/ line. Double click on it. The Files list in the right pane will update, and test_dump.gz will be the last file listed. Double click on this file.

The file dialog will close and the hustler(1) main window will come up.

If you don't want to use the file selection dialog, you can specify the input file on the command line. I.e., from the top of the shoki source tree:

	# /usr/local/shoki/bin/hustler -r test/test_dump.gz

...would accomplish the same thing as the exercise with the file selection dialog.

1.2 Interface overview

The default hustler(1) view looks like this (click on the image for a larger view):


Starting at the top of the screen, you'll see:

The filename of the input packet dump
Using all 1386 packets (1 filters unused)
The number of packets displayed. Ignore the `1 filters unused' part for now.
All packets
The current selection criteria. When you start up hustler(1) this will always be `All packets'. We'll look at selection criteria later.
0, 0, 0
The current coordinates of the cursor. As you move the cursor around in the hustler(1) window this will be updated.

The view panes are, counterclockwise from the upper left:

X-Z view
This is a plot of the destination port versus packet number for each packet in the input dumpfile. The packet number is simply identifies the order in which the packet appears in the dumpfile.
Note that the horizontal axis is the same as the horizontal axis of the pane below, and that the vertical axis is the same as the horizontal axis in the lower right.
X-Y view
This is a plot of the source port versus the packet number.
Again, note that the horizontal axis is the same as the horizontal axis of the pane above, and the vertial axis is the same as the vertical axis to the right.
Y-Z view
This is a plot of the source port versus the destination port.
Note that the horizontal axis is the same as the vertical axis of the pane to the upper left, and the vertical axis is the same as the vertical axis of the pane to the left.
Isometric view
This is a three-quarter view of a 3D model of the three axis displayed in the other panes.

In order to visualise what these panes represent, imagine the x-y view as the side of a box facing you. The x-z view would be the view looking down into that box from above, and the y-z view would be the view looking in from the righthand side. In other words, if you printed out a capture of the main window, and cut it out (removing the isometric view), then you could fold the L-shaped result into three sides of a cube.

The fact that all of the axis labels appear in the lower left corner of their respective panes can be a little confusing at first. Once you get used to the layout of the panes (or are already familiar with this kind of layout, i.e. from drafting) you shouldn't have any problems. Just remember that the label doesn't necessarily correspond to the axis it lies along: the label in the x-z pane (upper left) is for the z axis; the label in the x-y pane (lower left) is for the x axis; and the label in the y-z pane (lower right) is for the y axis. You'll notice that one of the scales alongside each view will change colours when the mouse pointer is in that view pane. The scale that highlights corresponds to the labelled axis. Note that one line of the bounding cube in the isometric view will also be highlighted. This line will correspond to the location of the scale in the isometric view.

1.3 Navigation

1.3.1 Axis Properties

Move the mouse pointer over one of the 2d panes in the main window. Right click and hold to bring up the popup menu, and select the Axis Properties submenu. It contains:

Toggle Active
This tells huslter(1) to use this axis variable for clustering. If an axis is inactive then hustler will ignore it. If an axis is marked as active, its label will appear in white; if it is inactive, the label will be in red.
Toggle FFT
If selected, the data for the selected axis variable is FFT transformed, filtered, and then inverse transformed. What this is useful for is discussed in Section 2.1 of this manual.
Toggle Log
If selected, the variable will be displayed on a logarithmic axis. This is useful for variables like source and destination port, where you have a bunch of points clustered down below 1024, and then the rest randomly scattered between 1024 and 65535.

Note that this property is for display purposes only; it has no effect on the underlying data.

Move the pointer to each of the 2D panes, right click, and toggle the axes to active. When you're done, the label in the lower left corner of all three panes will be white.

1.3.2 Axis Variables

Although you can only view three variables at a time, hustler(1) can keep track of many more. Currently, all of the variables in the standard IP, TCP, UDP and ICMP headers are available, as well as several special variables.

Move the mouse pointer into the any of the 2d panes in the main window and click and hold the right mouse button. This brings up a popup menu. From this popup, select the Axis Variable submenu. It lists all of the available variables. When you move the mouse pointer into one of the 2d panes and select a variable from this submenu, that variable will be mapped onto the current axis. So if you moved the pointer into the lower left pane it would change the x axis variable; in the lower right, it would change the y variable; and in the upper left it would change the z variable.

Go ahead and move the mouse pointer into the lower right pane, right click to bring up the popup, and select Axis Variable->IP Identity. This should change the x-y (lower left), y-z (lower right), and iso (upper right) views. The x-z (upper left) view will be unchanged; you just modified the y variable, and nothing in the x-z pane is dependent on it.

The y axis label is red. This means that it isn't active---if you conducted any clustering operations (which we'll get to in a bit), they wouldn't be affected by the IP Identity values. Reselect source port as the y axis variable (Axis Variable->Source Port with the pointer in the lower right pane). The label is white again. Note that the currently selected axis variables in the display have nothing to do with what axes are marked as active for clustering operations; an axis is not activated by viewing it, and it isn't deactivated when it isn't being viewed.

1.3.3 Clustering

The clustering functions are designed to group packets together based on user-specified critera. In this section we'll go through some very simple examples to get an intuitive feel for how this works.

First, toggle the Packet Number and IP Identity axes to be active (all the others should be inactive now).

Select Packet Number as the x axis if it isn't already (lower left pane), IP Identity as the y axis (lower right pane), and Destination Port as the z axis (upper left pane). The Destination Port label should be red (because it's inactive), but the others should be white. Toggle the y axis to be a non-log axis.

You should now be looking at four views of a big fuzzy wall with two line segments in front of it and two line segments behind it (and a couple random dots elsewhere). The wall fills the entire x-y plane, and represents a more or less random scattering of IP IDs. The line segments represent a stream of packets with identical or incrementing IP IDs. But we're not really concerned with what the data actually represent; we're just going to take a look at how the clustering widgetry works. Since we've set things up with only two active axes, we're going to be looking at how 2d data is clustered first. Concentrate on the x-y view (lower left); since only the x and y axes are active, they're the only ones that will affect clustering.

Go ahead and right click in any of the 2-d panes to bring up the popup menu. Select the Clustering submenu from the popup, and then select Fast Radix Clustering. The dots should change colour. Most of them should be a single colour (i.e., green), and a couple (mostly along the left edge of the x-y view) will be white. If you're having trouble making out the different colours, you can try adjusting the point size (right click, select Point Properties->Size from the popup menu, and then pick a point size from the submenu).

With the mouse pointer in one of the 2d panes, click the left mouse button once. The white points should turn grey and one of the status bars at the top of the screen (third from the top) should update to read `Cluster 1 of 1 (1346 packets)'. This tells you that you've currently got cluster 1 highlighted (in this case it's the only cluster), and it contains 1346 packets.

Left click again. This will select the unclustered packets (there should be 40 of them) and grey out the clustered packets. Left click one more time, and you'll be back to where you started. Left clicking highlights the next cluster. If you've got the last cluster selected, it will highlight the unclustered packets, and if they're selected, it will go back to the default view (everything selected).

1.3.4 The Menu Window

The hustler(1) main window always displays three axes. The underlying data model keeps track of many more. If you want to check or change the properties of any of these axes, you can select it as the axis variable in one of the display panes. This can be tedious if you're tweaking several axes.

The easier way to check or to make changes to many axes at once is to use the menu window. Right click on one of the 2d display panes to bring up the popup menu, and then select Open Menu Window.

A new window will open, containing a collapsed tree view. Click on the triangle to the left of the Axis Properties label. This will expand the tree view. The result should look like this:

[Menu Window]

Scroll down slightly to locate the Destination Port line. If you've been following along in this tutorial, Destination Port should be the current axis in the upper left (X-Z) 2d display pane.

Click on the Log checkbox (it's the centre one) in the Destination Port row. The view in the main window will update to reflect the change. You can also toggle the Active and FFT properties from here. This is equivalent to selecting the corresponding item under the Axis Properties submenu from the main window popup.

2. Interface Reference

2.1 Axis Properties

Toggle Active
Marks the current axis as active. Active axes are used in clustering, inactive axes are not. All axes are inactive by default.
Active axis labels are displayed in white; inactive axes are labelled in red.
Toggle FFT
Values for the current axis are FFT transformed, filtered, and then inverse transformed. Selecting Toggle FFT again will undo the filtering.
Only values for currently selected packets are affected.
Toggle Log
Toggles the current axis between logarithmic and proportional plotting.
This has no effect on the underlying data; it only transforms the display.

2.2 Axis Variables

Selecting a variable name from the Axis Variables menu will set the current axis to that variable.
This only affects the display; the underlying data model is unaware of which axes are being displayed.

2.3 Clustering

Reset Clusters
Discard any clustering information. Will not affect things like active axes or metric tensors.
Fast Radix Clustering
Pass the currently selected packets through the radix clustering algorithm.

2.4 Config

Load Config
Save Config
Load or save (respectively) the current axis settings and other clustering-related variables.

2.5 Filters

If multiple shoki filters were given to hustler(1) prior to reading the input data, individual filters may be selected from this menu. Selecting a filter will highlight the packets matching that filter. Nonmatching packets will be greyed out.

2.6 Point Properties

Toggle Smoothing

Selecting this option will toggle OpenGL point smoothing.

Turning smoothing on is generally desireable when attempting to distinguish features among tightly-packed packets. Having smoothing on may slightly degrade performance.

Smoothing is off by default.


Selecting a point size from the Point Properties->Size menu will change the size of the dot rendered for each data point. This has no noncosmetic effects.

2.7 Open Menu Window

Selecting this option from the popup will open the menu window if it is not already open.

The menu window contains a list of the axes of the hustler(1) data model. The metric for each axis may be edited from the menu window, and the active, log, and FFT properties may be toggled.

2.8 Selection Mode

Selection modes affect the mouse button semantics in the 2d display panes.
If the data is currently clustered, the left button will cycle backwards through the clusters and the middle button will cycle forward through them.
The currently selected cluster will be rendered in colour, and the others will be greyed out.
If the data is not clustered, the left and middle buttons will do nothing.
In both the Resize and Select modes the left button will handle selection. The first click will specify the first corner of the selection area. The second click will specify the final corner of the selection area. Middle clicking at any time will reset the selection area.
In Resize mode, selecting the final corner of the selection area will zoom the 2d panes to display the selection area. Selecting a different mode will unzoom the display panes.
Selection areas are persistant between modes, so you may select a region in Select mode without resizing, and then switch to Resize mode to zoom. This may be helpful in fine-tuning selection areas.

2.9 Phase Space

The only option under the Phase Space menu is Open Window. Selecting it will render a phase space plot in a new window. The variable used in the plot will be the axis variable of the current window.

2.10 Selection

Selecting Reset Selection will undo any current selection area.

2.11 View Packets

Currently, selecting Save Current will cause any packets matching the current selection criteria (filter, cluster, or selection area) to be output to stdout.

[Shoki Homepage] []