Pp. 119-130 of the Proceedings

Abstract

Thresh is a simple SNMP [1] monitor, written in Scotty [2] (Tcl [3] with the Tnm extensions), which uses the UNIX file system hierarchy for configuration and data storage. Thresh compares SNMP variables to per-device thresholds or values, and issues notifications if any current SNMP variable values are unacceptable (or unexpected). Thresh can be used by itself, or as a complement to other network management and monitoring tools. Thresh can be thought of as fitting in between tools that generate immediate emergency alerts (such as Big Brother [4]) and trending and history tools (such as Cricket [5]).

Introduction

Virtually every computing system and network has some kind of monitoring mechanism in place these days, but in some cases the monitoring system consists only of users phoning and asking if there is something wrong with the network. For those interested in something a little more advanced or automatic, there are quite a few software packages available that do some form of monitoring. They range from the very simple (ping tests, etc.) to the very complex (large commercial packages that map networks, configure devices, and make your lunch), with various alternatives in between.

In the realm of ``simple'' monitoring software, packages can generally be divided into two types:

• Alarmers - Software that monitors connections, services, and so on, and sends out an alarm (mail, pager, smoke signal) as soon as it detects a problem. Some examples of alarmers are Big Brother [4], nocol [6], and Spong [7].
• Trenders and Trackers - Software that keeps history or trend data for future analysis or review. The most obvious examples here are MRTG [8] and Cricket [5].

A couple of years ago it occurred to me that there was another class of monitoring that didn't seem to be very well addressed - low to medium priority tracking of certain parameters and their values on computing systems and network devices. For example, you might want to track configuration changes, system or device reboots, or network interface status or change time. These are things that you may want to know, but which don't necessarily indicate an immediate problem and which may not be worth waking anyone up to investigate.

Thresh was created to provide this kind of monitoring. It tracks SNMP variables, compares them to threshold, pre-set, or last-observed values, and reports (typically via email) unexpected changes or out of range values.

Why Use Thresh and Not Something Else

I've become convinced that, ugly as it may sometimes seem, the Simple Network Management Protocol (SNMP) [1] should be the basis for just about every monitoring system. Virtually every network connected device either comes with an SNMP agent or can run one with only a modest amount of effort. Most SNMP agents can provide a vast amount of (usually) useful information, and most agents for general purpose computers can be extended to provide just about any data that you might want to have.

Some monitoring systems (such as Big Brother and Spong) rely, to a greater or lesser extent, on separate client agents, running on each system that needs to be monitored, with a system-specific reporting protocol between the agent and the management station. This approach can be somewhat limiting (it's hard to use on things like networking equipment for example), can result in some duplication of services (if you need an SNMP agent for other purposes), and limits both what you can do (they're often not extensible), and where (as your firewall may not be able to pass the particular protocol implemented by the software). Thresh avoids these kinds of problems by using only SNMP for communication. [Note 1]

It's often useful to be able to track certain SNMP variables, but you don't always need to know about changes immediately - it's often good enough to hear about them the next time you read your mail. For example, the system.sysUpTime.0 SNMP variable gets reset every time a system or device gets rebooted - if you get notified every time that variable resets, you'll know if you've got a device reliability problem (or an extension cord that people keep tripping over). Similarly, you can generate disk capacity threshold warnings, network bandwidth warnings, network interface up/down notices, and so on.

With many monitoring systems, this kind of low-priority information can be hard to generate. Many of the most common freely available packages tend to have only a small number of notification or alert mechanisms, and are built with the idea that you're monitoring vital services and networks - sometimes everything is assumed to be an emergency.

Thresh was created to address this kind of medium-level monitoring need.

There are a number of commercial monitoring systems available. Some, such as Spectrum [9], HP OpenView [10], and NetCool [11], are widely deployed and very well respected, and most of them can do (or can be made to do) most or all of what thresh does. However, the commercial packages tend to require a much larger monitoring infrastrucutre, and a much larger committment of time and money to implement, operate, and maintain. For small to medium sized sites, small, simple monitoring tools, like thresh, are often the best choice.

Implementation

Thresh was implemented in Tcl [3], using the Tnm network management extensions provided by the Scotty/Tkined [2] software. The Tnm extensions are a toolkit of Tcl procedures that make it easy to perform SNMP operations.

Tcl was chosen because it is well suited to this kind of task, and the Tnm extensions provided just the right functionality. At the time thresh was first contemplated, the SNMP modules for Perl were relatively primitive when compared to Tcl and Tnm.

Thresh has benefitted from the use of a scripting language, and the string and array manipulation routines provided by Tcl. Tcl allows the use of an interactive ``shell'' for testing and development. Being a traditionalist, I tended to develop incrementally, using the well-known edit, run, repeat cycle, rather than a more ``modern'' approach to program development. Thresh is currently about 700 lines of Tcl.

In retrospect, I'm still glad to have chosen Tcl in preference to Perl, C, awk, or Visual Basic.

Simplicity

One of the design and implementation goals for thresh was simplicity. That goal has been addressed in the following ways:

• One main program, thresh, and a very small number (currently one) of simple utility programs.
• Implemented in a well-designed scripting language (Tcl [3]), using a well-known and effective set of library routines (Scotty).
• Simple to run - requires no additional daemons (just a crontab entry), and no addtional software is required on remote systems [Note 2].
• Requires no special userids or groups. (It would probably be useful to have a separate userid to run thresh, and perhaps a userid or group to own the config files, but that decision is left up to those installing and using the software.)
• Configuration is relatively straightforward, and involves only two simple file formats. The hierarchical configuration structure makes implementing default settings easy and flexible.

verbose = true
name = mydomain.net
community = hello
mib = /usr/local/mibs/ascend.mib
# big network, long timeout
timeout = 20
notifier = threshmail jsellens
syslog = local1.info

# this is a comment
S system.sysDescr.0
S system.sysContact.0
I system.sysUpTime.0
C interfaces.ifTable.ifEntry.ifDescr.5
C interfaces.ifTable.ifEntry.ifAdminStatus.5
C interfaces.ifTable.ifEntry.ifOperStatus.5
G ucdavis.memory.memTotalReal.0                   90000
G enterprises.ucdavis.memory.memAvailReal.0       3000
L loadTable.laEntry.laLoad.1                      1.20
L loadTable.laEntry.laLoad.2                      1.50
L loadTable.laEntry.laLoad.3                      2.00
V snmp.snmpInPkts.0

<ws>VARNAME<ws>=<ws>VALUE<ws>
<ws># comment ...
<ws>