Our first metric of performance of a disconnection policy is,
expectedly, the number of busy signals that the policy incurs for a
given number of modems. Nevertheless, this number depends primarily on
the number of users who can potentially be disconnected and the latter
number is almost the same for all replacement policies. Our approach
is to have an inactivity threshold (or timeout), 
. Any
user who has been idle for more than seconds can potentially be
disconnected. The difference between different policies is in
which user (or users) actually do get disconnected. Thus, the
number of busy signals is similar across different policies that all
have the same value of . (The numbers are not identical because
the set of users that can be potentially disconnected depends on which
users were disconnected in the past.) Therefore, the number of busy
signals is a good indicator of the overall quality of service, but it
is not a good differentiator of the various disconnection policies.
Our other performance metrics are identical to those used in the work
of Douglis and Killian. These metrics attempt to measure user
``inconvenience'' due to disconnections. The approach consists of
setting a threshold 
. If a user is disconnected and does not
become active again within seconds, then the disconnection is
deemed successful and is termed a soft fault. If, however, the
user becomes active within seconds, the disconnection is
considered a hard fault (or just fault). (Hard faults
were called ``bumps'' in the Douglis and Killian study, a term
borrowed from the disk spin-down domain [DKB95].) The
severity of a hard fault is a linear measure of how close the
user's idle time after disconnection was to the threshold . (That
is, severity is a linear function of idle time after disconnection,
such that an idle time incurs severity 0, and an idle time 0
incurs severity 1.) Two metrics that we used for evaluating
disconnection policies are the number of hard faults and the severity
of hard faults. Nevertheless, the results using these two metrics were
very similar, with the fault severity slightly accentuating the
differences between disconnection policies. Since the two metrics
yield very similar results, as well as due to lack of space, we will
only use the number of hard faults as a metric in the following
section.
The reason for considering only a few of the faults to be significant has to do with the perceived inconvenience of disconnections by the user. A user is wrongly disconnected if her network connection is idle but she is actively using the machine and expects to be connected (e.g., the user may be writing an email reply off-line, which she will later send). In this case, the user will notice the disconnection and reconnect explicitly. This should be counted as a fault by the system because the user was inconvenienced and the lack of network connectivity was immediately noticed. On the other hand, if a user is idle and stays idle after a disconnection, the user was probably truly inactive. In this case, the inconvenience of reconnecting is much less and the user is likely to consider the disconnection justified.
Distinguishing between hard and soft faults may seem strange to readers used to replacement problems. We believe, however, that this metric is truly appropriate for the domain. Additionally, none of the results we present change qualitatively if we consider the total number of faults as a metric. That is, if disconnection policy A is better than disconnection policy B using the number of hard faults as a metric, then A is also better than B using the number of total faults as a metric. What changes is the quantification of how much better A is.