musings

by Rik Farrow
<rik@spirit.com>

Rik Farrow provides UNIX and Internet security consulting and training. He is the author of UNIX System Security and System Administrator's Guide to System V.

Ah, if life were only that simple!

The world we live in is a dangerous place, and it just got a bit more unpleasant. Of course, our daily travails are nothing compared to those of people living on those islands in Indonesia where the Christians and Moslems take turns killing one another, or in the Russian province of Chechnya, which perhaps will have settled back into relative calm by the time you read this.

But even in the Western world, there are dangers to face. For example, I set out on a journey recently. The first phase of that journey entailed hurtling through the desert at 110 feet per second (about 33.5 meters per second). Then I entered an area known to be dangerous. Armed guards were stationed at strategic locations, and abandoned vehicles were quickly towed away lest they explode unexpectedly. Everyone entering the facility was searched for guns and explosives, and there were frequent announcements that any "unusual or suspicious" behavior should be reported immediately.

I am talking about going to the airport, of course. Many of us take things like driving on crowded freeways and traveling in airplanes (where a single defective part might send us diving to our deaths) as part of everyday living. I admit this is a great advance over the "good old days," when living to 45 was considered a great feat, and any travel at all took many times longer, with a much greater risk. Still, it is no wonder that we all feel a little stress at times.

And just this week as I write this, the second week in February, something long dreaded by many of us in the security industry has come to pass. As if ordinary network-based denial-of-service (DoS) attacks were not enough, distributed DoS attacks using hundreds of agents have begun taking down e-commerce sites.

Not New
For some people, this is not news at all. Distributed DoS attacks were the topic of this year's first CERT advisory, and this advisory actually preceded widespread attacks. But there had been attacks. Attackers targeted an IRC server at the University of Minnesota last summer, disabling not only that server but also most of the university's network infrastructure for two days. During this attack, over 2,000 different remote systems participated in sending floods of UDP packets. The University of Minnesota has an OC3 connection to the Internet, and another OC3 to Internet2, and both of these links remained swamped for days.

The only "bright side" to this attack is that the tool used, Trinoo, does not spoof source addresses, making it relatively easy to find the agents sending the floods. But it was the scale of the attack that was mind-boggling. John Ladwig, security architect, Networking and Telecommunications Services, wrote me saying that "it frightened me that someone would 'throw away' approximately 2,000 compromised hosts, primarily very well-connected [and] fairly powerful ones, presumably to seize IRC channels." This happened in August of 1999, and by February 2000 the attacks had a new level of visibility, even gaining the attention of Attorney General Janet Reno and spots on all the major news sources. Reno promised to prosecute the offenders, something I found interesting. Current US law does not include denial-of-service attacks, but covers only "unauthorized use." Since these attacks come from systems that have been broken into, the perpetrators can be prosecuted because they broke into systems, but not for the denial of service.

There are other attack tools — Tribal Floodnet, TFN2K, and Stacheldraht — all of which can spoof source addresses. If you are the victim of one of these attacks, it may not be immediately obvious that addresses are spoofed, because they will be coming from many different sources. Even if the addresses are not spoofed, you will need the cooperation of many remote sites to stop the attacks. In many cases, ISPs' systems were used as agents, as these systems are not only usually powerful UNIX systems, but by definition are connected to relatively fast network links. Another "bright side" to the recent attacks is that most of them come from the continental United States, against other U.S. targets, so the owners of the attack daemons and the victims both live in similar time zones and speak English. Just imagine what fun you would have trying to convince a Russian or an Indonesian network administrator that some system at her site has launched a devastating attack on you. (Actually, if you have ever experienced handling situations like this, please contact me.)

Dave Dittrich of the University of Washington wrote several very useful analyses of these attack tools that you can download from his Web site: <http://staff.washington.edu/dittrich/misc/[trinoo.analysis|tfn.analysis|stacheldraht.analysis]>. If the tools have not been modified, you can use Perl scripts written by Dave to search for daemons within your own networks. Some of the daemons are programmed to respond to certain requests, and his Perl scripts trigger these responses — again, only if the daemons have not been modified so that the requests are now invalid. Also, if you are interested, you can download the attack tools themselves (unless the Feds have blocked this) from <http://www.technotronic.com/denial.html>. David Brumley, who works on security at Stanford, has written a tool for detecting daemons that will compile on Solaris and Linux: see <http://www.theorygroup.com/Software/RID>.

The various intrusion-detection vendors have added capabilities to detect master-daemon communications, and some scanners include the ability to send the triggering requests that will flush out daemons hidden in your networks.

The terminology I am using comes from the CERT advisory, <http://www.cert.org/advisories/CA-99-17-denial-of-service-tools.html>. Masters are the systems used to control the daemons, the software that actually does the flooding. The attacker connects to the master — sometimes using a special client, other times just by telneting to a port and entering a password — and uses the master to send commands to the daemons. TFN2K daemons run as root and sniff the network, looking for the encrypted commands that come from masters. No responses are sent back to the masters for routine commands, so there is no two-way communication that can be sniffed and used to discover masters. TFN, TFN2K, and Stacheldraht can all use ICMP packets for communication.

Masters and daemons sometimes have lists of addresses of their counterparts either built in or located on the local system, for example, in a file named ... [three dots]. If you find masters or daemons, don't just delete them! They may contain information leading to the discovery of still more exploited systems. Again, Dave Dittrich's papers have information on recovering lists of systems (which are sometimes encrypted, but with a key that can be recovered from the executable). If you do not wish to deal with this yourself, try CERT.

Are You Vulnerable?
On a barely related topic, the Chinese launched their own attack against U.S. government sites after the accidental cruise-missile bombing of their embassy in Belgrade in the fall of 1999. The Chinese attacks revealed that they had previously compromised over 4,000 systems, installing backdoors for later access and use.

I've mentioned over 6,000 compromised systems so far in this article, an infamous number. There were 6,000 systems exploited by the Internet Worm in November of 1988, at that time a large proportion of Internet servers and gateways. Today, 6,000 systems is barely a handful of the servers connected to the Internet. But the real question is, "Why are all these systems vulnerable?"

Distributed DoS attacks would not succeed without a ready supply of exploitable systems. Dave Dittrich's papers, and conversations with David Brumley of Stanford Computer Security, reveal the manner in which these systems are compromised. The attacker uses scanning tools that search through large ranges of IP addresses looking for vulnerable systems with tools like mscan and sscan. Then the attackers break into one of these systems and launch a script that uses the same exploit to break into a list of other systems. The exploit involves using rcp to the script-running system to download the agents and launch them. Brumley wrote "Often we'll find a list of hundreds of compromised hosts (usually because an intruder is rcp'ing over a rootkit and the rcp's are logged in SYSLOG) on another site that the administrator was ready to just delete!" More reasons to look before you wipe that compromised system clean.

And why are there thousands of systems waiting to be compromised? Usually because existing security patches have not been installed. I wrote a "passionate plea" in my last column about choosing one or two good ways of accomplishing this and other tasks, instead of inventing new ways for each LISA conference.

Another reason has to do with the lack of firewalls. Many sites avoid firewalls because they have an open policy. If all the firewall did was enforce RFC2267, that would help a great deal. RFC2267 describes ingress and egress filtering, techniques for blocking packets that have spoofed source addresses. Unless your site is a transit site — that is, you carry traffic for all parts of the Internet — you know exactly what source addresses should appear in packets leaving your site. By blocking packets that do not have one of those source addresses, you can prevent source address spoofing from your own site.

RFC2267 was written in response to the spate of SYN flooding that occurred after people unraveled the attack on Tsutomu Shimomura back on Christmas Day, 1996. SYN flooding became very fashionable among small ISPs that were attempting to damage competitors' reputations by flooding their Web sites and filling the TCP connection queues. RFC2267 is even more relevant today, with the increasing popularity of DoS attacks, where spoofing the source address makes sense and is easy to do.

Firewalls could also be used to block the widespread scanning that takes place. Or perhaps just to block the "r" commands (or does your site routinely trust everyone on the Internet, as was done in the old default configuration of Sun workstations)?

Force
If the recent attacks have continued, perhaps we will soon be forced to impose RFC2267 on our networks. In my opinion this would be a good thing, and it might come about in any of several ways. One way would be for the Network Access Providers (NAPs) to deny connectivity to sites that do not apply RFC2267 filtering. While tedious to set up (for larger sites), and requiring still faster and more expensive routers (or firewalls), this is nothing more than a good neighbor policy. That is, we will be good neighbors and not pollute the Internet with spoofed source packets from miscreants within our own networks. University of Minnesota does RFC2267 filtering all the way out to the edges of its networks, which encompass 50,000 systems. Stanford University filters at its borders.

Another way this might come about is through legislative action, but I am not holding my breath for that one. Perhaps the most likely, and the least pleasant, will be the civil lawsuit. eBay will sue one or more large universities or organizations as being the source of the floods against their Web servers that cost them hundreds of thousands (millions?) of dollars in possible revenues. If a suit like this were successful, or just long-lasting and unpleasant enough, network administrators would suddenly find themselves with the resources to make their networks more secure. That is, due diligence would be required, or the organization would face a lawsuit.

The icing on this cake came when CERT put out its second advisory of 2000, warning that it is possible for evil sites to include information in links that can cause scripts to be executed on clients visiting trusted sites. The advisory (<http://www.cert.org/advisories/CA-2000-02.html>) suggests that Web-site operators revise any scripts that create forms to filter out HTML that may have been added by a potential attacker. This advisory also suggests disabling all forms of active scripting (Java, JavaScript, and, by implication, ActiveX started by JavaScript), something that many of us have already done. Sorry, Microsoft, but you will have to come up with another way of world domination. Perhaps giving away Windows 2000? In late February, a version of the trinoo attack tool appeared as a trojan horse installed along with Back Orifice on Windows systems, so UNIX systems are no longer the only host for attack daemons for DDoS.

Sigh. The world is a dangerous place. Still, I am glad that my neighbors keep their AK47s unloaded and put away, and that no one is launching artillery or rocket attacks against my neighborhood.