by Chris Lalonde
<lalonde@ingenia.com>

Chris Lalonde is the senior security consultant at Ingenia Communications. He specializes in network penetration testing, firewall design, and software auditing.

Network Flight Recorder

Network Flight Recorder (NFR) is the latest venture by Marcus Ranum, security guru of the first order, and his associates. NFR is, simply put, a network sniffer and processing language that hooks into a Java GUI to display results. Data from the sniffer is saved to files that the Java GUI allows you to query.

Anyone who has been around the back end of a network late at night, either trying to fix the network or trace an intruder, knows the value of network sniffers. These are the debugging tools of choice for the network administrator, and historically they have been divided into two categories: network monitor/statistic-gatherer for use by network administrators, and sensitive data sniffer for use by network intruders. NFR can be used in place of both tools and for a variety of other network administration jobs.

The beauty and beast of NFR is that it is really a programming language for a network sniffer and display program. With NFR, you can extract any data off a TCP/IP network. But there is a catch: you probably have to write the program yourself. Future versions of NFR will support other network protocols as well.

So why bother with NFR at all, you ask. The answer is easy: the language used (known as n-code) is pretty simple to pick up for anyone who's programmed in C or Perl. NFR now gives you a single interface, so you can finally put away all those shell scripts and tcpdump hacks you were using before. Finally, you have one tool to do all the network sniffing and analysis that you want.

Background

NFR consists of four major pieces:

1. packet sniffer, the program that actually listens to the network

2. nfrd the daemon that manages and executes the n-code (filters), which is the heart of NFR

3. back-end processors, which do the dirty work of actually taking the output of the filters (n-code programs) and recording the data into files that can then be queried (Currently only two types of data file formats exist: list and histogram.)

4. GUI, A Java-based interface that allows you to query the files created by the back-end processors

An alert daemon (alertd) and space management daemon also exist, but they are not really part of the NFR core. The alert daemon is used to send alerts (email, pager, fax, or print) when something really big happens (you can define what really big is), and the space management daemon is there to rotate data files and prevent your hard drive from filling up if you are gathering a lot of data.

These four pieces work together basically as follows. Data are captured by the sniffer and passed to the nfrd. Nfrd checks the packet against currently installed filters to see if it should be passed to a back end. If the data should be captured, the back end records the data to a file. Finally, when the user enters a query through the GUI, the data are retrieved from the appropriate file and displayed accordingly.

According to the folks who make it, NFR is supposed to be a sort of Swiss army knife for the network administrator. The idea is that no one but you can really know what tool you'll need today. So this generic tool is highly adaptable to the needs of a network administrator. What this really means is that NFR is supposed to be a highly programmable piece of software, which it is, and is not for the programming faint of heart or for those looking for a plug-and-play solution. What it also means is that if you understand what you want and can do some simple programming, you can make a wide variety of nifty tools for monitoring your network and its traffic.

Like much of the work Mr. Ranum has done, NFR is available to the public for noncommercial use at <http://www.nfr.net>. As always, check the license for details on acceptable use.

Competitors

NFR (today anyway) is not an IDS. It can do IDS-like things if you want to program it that way, but it can also do a multitude of other things that most IDS do not do; it all depends on how you program it. NFR does come with a number of "built-in" programs that are useful, including a SMTP monitor, a Web server monitor, and a number of protocol monitors that let you watch packet/machine statistics.

As far as I have seen (which may not be that much), there are not any direct competitors to NFR; it is just too generic and flexible a tool. However, for those looking for IDS tools, the market is getting larger.

Some IDS products are: Real Secure by ISS <http://www.iss.net>, Session Wall-3 by Abirnet <http://www.abirnet.com>, and NetRanger by Wheelgroup <http://www.wheelgroup.com>

My NFR Experience

My network (I call it that though many others have and do work on it more than I do now) consists basically of four segmented class C networks and an internal unroutable class A network. The external networks have been separated into two specific zones: one for our clients and one for us (the good guys). Between each network lies a firewall with various rules. The firewalls are swappable and can be updated from a central point over a secure link. So if one firewall fails, we can always swap a backup in place in about five to ten minutes of failure. We believe in the old security-in-depth principle. Did I mention being paranoid ?

Because we are currently evaluating NFR, we have only one host to compile, run, and test n-code applications. NFR is nonintrusive, so it can sit and watch the network, carrying out three tasks:

1. making sure that my firewalls are doing what they are supposed to be doing (i.e., a firewall consistency checker)

2. watching inbound traffic for what I consider "bad traffic" (e.g., http requests for phf, popmail requests, etc.)

3. gathering network statistics on type and volume of traffic throughout the day

In the past, I have kludged together the invaluable tcpdump and some Perl scripts to regularly update me on the "health" of my network. I currently am replacing most of those "hacks" with n-code programs and then comparing how they perform. The main advantage so far has been that NFR enables me to have a single interface and log and alerts for all of the programs I was using before (NNstat, tcpdump, etc.).

After getting approval (isn't that always the fun part), my plans involve setting up multiple NFR hosts on separate segments and having those hosts "ship" back the data to a central host via a secure channel. Each host will have the capability of sending email and paging an on-call administrator if a critical "error" occurs. This way I hope to obtain the redundancy and network coverage of having multiple hosts while maintaining a central point for logging and statistics gathering.

Some simple examples of code

To write a program in n-code you must have three basic files: program.nfr, program.cfg, and program.desc.

Programs (a.k.a. filters) are usually placed in a directory to group programs together, so programs test1, test2, and test3 go in the test directory, etc. To ensure that this directory structure is passed on to the interface, a config file for each directory (i.e. group) is created.

For our test code, I'll make a test directory and create a test.cfg file. So if I do an ls I have the following files:

packages> ls
test/
test.cfg

and the test.cfg file looks like

packages>cat test.cfg

###############################
#
# Test
# By C.Lalonde Ingenia
# Communications Corp
#
enabled=true
# allow NFR to start itself
title= Test ver0.0
# the title that will appear
#end
###############################

This is the base config I use when testing some new application. Comments in all files have a "#" sign in front of them. You don't have to include so many comments; I'm just being a good coder for this one example.

Once you have a directory and the directory config file, you now have to create the three files needed to create a filter. So if I cd into the test directory shown above and do an ls, I'll see the following:

packages>cd test
packages/test>ls
test1.nfr
test1.cfg g test1.desc

Files ending with a .nfr extension are the actual nfr code: the guts and glory. Files ending with .cfg extensions supply the interface configuration information for each program, and files ending with .desc are plain text descriptions of what your code is supposed to do.

NFR is like C in its semantics but like Perl in its variable implementation. .nfr programs are loaded at run time and only actually "work" when a "trigger" is pulled. A trigger can be almost any network event, from all TCP traffic to a highly specific event (e.g., when you see the string phf coming from a client to a server on port 80). Once set in motion, the program can record all the data and do other nifty things, like send email or call a pager.

Here is a simple program called test1.nfr that records all TCP traffic:

packages>cat test1.nfr

##################################################
# test1
# By C. Lalonde Ingenia Communications Corp
##################################################
#
# Test program to see what this puppy can do
#
# Jan/26/98
##################################################

test1_schema = [ 1, 1, 1, 1, 6, 6 ];
filter test1 tcp( ) {
 record packet.sec, tcp.sport, tcp.dport, ip.src,
   ip.dst to RECORDER_TEST;
}
RECORDER_TEST = recorder ("bin/list
   packages/test1/test1.cfg",
  "test1_schema");
#
# end
##################################################

Now I'm going explain what the above says, line by line.

test1_schema = [ 1, 1, 1, 1, 6, 6 ]:
­ A schema describes the data types to nfrd and how they should be recorded, in this case:

1 - int - system.time
 1 - int - packet.sec
 1 - int - source port
 1 - int - destination port
 6 - ipv4 addr - source ip address
 6 - ipv4 addr - destination ip address

filter test1 tcp(){

­ Begin test1 filter recording tcp packets.

record packet.sec, tcp.sport, ip.src, tcp.dport,
 ip.dest to RECORDER_TEST

­ This is the record statement. In this case, I'm recording a hash of the tcp connection, the source ip port, the source ip address, the destination ip port, and the destination ip address to the recorder RECORDER_TEST.

RECORDER_TEST = recorder ("bin/list packages/
 test1/test1.cfg", "test1_schema")

­ This is the definition of the variable RECORDER_TEST. In this case, I'm saying that RECORDER_TEST is a recorder of type list, and NFR should check the files test1.cfg for details on how to show the data sent to this variable and test1_schema for details on how to record the data sent to this variable.

Now that we know the basics of the .nfr file, we'll move along to the .cfg files. As we have seen, the .cfg files tells the nfrd how to display the information passed to it. A simple config file looks like:

packages/test>cat test1.cfg

################################################
#
# By C.Lalonde Ingenia Communications Corp.
#
################################################
# Test1 v0.0
#

enabled=true
# allow nfrd to load this

gui=list
# what type of gui should we display (two types
# are list and histogram) we've said that data
# would be saved as type list in the test1.nfr
# file so we have to say list here

num_columns=4
# number of columns of data we're going
# to display

# variable types for each column in the display
column_1_type=p_src_port
column_2_type=p_dst_port
column_3_type=p_src_ip
column_4_type=p_dst_ip

# titles you will see for each column in
# the display
column_1_label=Source port
column_2_label=Destination port
column_3_label=Source IP address
column_4_label=Destination IP address

rollover=300
modified=true
origin=C.Lalonde Ingenia Communications Corp
title=Test1 ver 0.0
# title we're going to see

cfversion=1
# version number

rollover_size=YES
# are we going to roll over the data for size ?

rollover_size_val=1024000

# at what size do we roll it over
# pretty small since this is a test and we're
# recording everything

rollover_time=YES
# are we going to roll over the data for time ?

rollover_time_val=300000
# how frequently

archive_path=data/%p/%b/%y/%m%d/
# where to archive the data
#
# end
################################################

The .cfg file for a program is very similar to that of the directory .cfg file. The comments in the file should be enough to get you started.

Now we've got two of the three basic; the other file isn't really required, but it is usually included for completeness. So test1.desc looks something like this:

packages/test>cat test1.desc

################################################
This is a Test program which records all tcp

# packets.
#
# By C. Lalonde Ingenia Communications Corp.
#
# end
###############################################

The .desc file is simply a description file included to allow for explanatory comments.

Now we have all three files, and they are sitting in the correct directory. The best thing to do is quickly check your test1.nfr with the handy test-nfrd program. Simply type test-nfrd test1.nfr and if you have any syntax errors, the entire program will be displayed and the errors will be surrounded with >><< symbols.

Next start NFR as the documentation instructs, and check to see if it works.

A good place to start looking/learning about the programs is to review the examples included with NFR and have a read through the documentation. There are many excellent examples, and they are well documented.

Negative Aspects of NFR

packages>ls

test/
test.cfg

packages>mv test* ..

Packages can also be disabled via the GUI or changing enable=true to enable=false in the appropriate config file.

As with any new programming language, it is difficult to jump right in and start programming in n-code. Although NFR has made a large effort to aid beginners in overcoming their initial difficulties and hesitations, improvements could still be made. The NFR system and n-code are not for the faint of heart; but they will be much easier for those who understand network protocols and what data they want to record.

My biggest complaint about NFR to date is the documentation. Currently, there are several shortcomings, the biggest being topics that are not covered (e.g., the "schema= [1,1, 6,6,2]" definition is not covered anywhere except in the sample code, and there only briefly).

Secure Networks Inc. recently published a technical paper that outlines problems inherent in using Intrusion Detection Systems based on network sniffing. It outlines a number of problems with various IDS and once again shows that "there is no silver bullet." This paper is definitely worth reading. You can find the it at <http://www.securenetworks.com/papers/ids-html/>.

Positive Aspects of NFR

If you're interested in trying it out, subscribe to the NFR mailing lists. They give you direct access to the actual developers (what an amazing thing this is), and the response time has been excellent. You can actually talk to the developers and possibly have an effect on the product. This access to the developers has been key in overcoming the problems I have had with the documentation.

The potential for NFR on a network is huge. Why? Picture several connected NFRs providing a "web" of network analysis and security monitors at a fairly reasonable cost. Also, it would be custom built for your network to be tailored to your specific needs. NFR is easily expandable in that you could start with one or two NFRs and easily connect more as load and network growth require. Finally, all the "monitors" can be controlled and configured from a central point, allowing easy maintenance.

Conclusions and Future Directions

Whatever NFR's future, it is not an IDS, not exclusively anyway. You can make NFR an IDS, but that does not seem to be the only design goal. Additionally, with the increase of protocols and platforms supported (an NT version is due out sometime in the early summer, along with support for Novell, or so the NFR team hints), NFR continues to expand its usefulness on today's multiprotocol and multiplatform networks.

Capturing HTTP GET Requests

Here's a simple piece of n-code that records all http GET requests. All requests should take the form of:

Method SP Request-URI SP HTTP-Version CRLF

(e.g., GET /index.html HTTP/1.0 )

What we are telling NFR to do with the following code is to, first, look only at data sent or received on port 80, second, start recording only when you see the string GET, and, third, stop recording when you see the string HTTP/.

If you wanted to record all POST or HEAD requests, you would simply modify the piece of the code that says start: GET to start: POST etc.

This is likely the simplest method of recording request types or any string and can be a little error prone (because not every implementation of the HTTP protocol is the same).

>cat test2.nfr

##################################################
# test2
# By C. Lalonde Ingenia Communications Corp
##################################################
#
# record all http GET requests
# March/13/98
##################################################

test2_schema = [ 1, 1, 6, 6, 2 ];

filter urlcatcher tcp( port: 80, start: "GET",
stop:"HTTP/" ) {

# port: - record data from/to this port
# start: - start recording data when we receive
# this string
# stop: - stop recording when we receive this
# string

record packet.sec, ip.src, ip.dst, tcp.bytes to
RECORDER_TEST;

# tcp.bytes - where the recorded data is stored

}

RECORDER_TEST = recorder ("bin/histogram
packages/test1/test2.cfg", "test2_schema");

Note: As you can see from the placement of the .cfg file in the above recorder statement, I have not created another directory. I am simply adding another n-code application to the same test1 directory structure that I used for my previous example.

>cat test2.cfg

################################################
#
# By C.Lalonde Ingenia Communications Corp.
#
################################################
# Test2 v0.0
#
enabled=true
# allow nfrd to load this

gui=list
# what type of gui should we display (two types
# are list and
# histogram) we've said that data would be saved
# as type list in
# the test1.nfr file so we have to say list here

num_columns=3
# number of columns of data we're going to
# display

# variable types for each column in the display
column_1_type=p_src_ip
column_2_type=p_dst_ip
column_3_type=p_string

# titles you will see for each column in the
# display
column_1_label=Source IP address
column_2_label=Destination IP address
column_3_label=GET URI

rollover=300
modified=true
orgin=C.Lalonde Ingenia Communications Corp
title=Test2 ver 0.0
# title we're going to see

cfversion=1
# version number

rollover_size=YES
# are we going to roll over the data for size ?

rollover_size_val=1024000
# at what size do we roll it over
# pretty small since this is a test and we're
# recording everything

rollover_time=YES
# are we going to roll over the data for time ?

rollover_time_val=300000
# how frequently

archive_path=data/%p/%b/%y/%m%d/
# where to archive the data

>cat test2.desc

N-code to record all http GET requests

# By C.Lalonde Ingenia Communications Corp.