2006 USENIX Annual Technical Conference Abstract
Pp. 255–260 of the Proceedings
Securing Web Service by Automatic Robot Detection
KyoungSoo Park and Vivek S. Pai, Princeton University; Kang-Won Lee and Seraphin Calo, IBM T.J. Watson Research Center
Abstract
Web sites are routinely visited by automated agents known as Web
robots, that perform acts ranging from the beneficial, such as
indexing for search engines, to the malicious, such as searching for
vulnerabilities, attempting to crack passwords, or spamming bulletin
boards. Previous work to identify malicious robots has relied on
ad-hoc signature matching and has been performed on a per-site
basis. As Web robots evolve and diversify, these techniques have not
been scaling.
We approach the problem as a special form of the Turing
test and defend the system by inferring if the traffic source is human
or robot. By extracting the implicit patterns of human Web
browsing, we develop simple yet effective algorithms to detect human
users. Our experiments with the CoDeeN content distribution network
show that 95% of human users are detected within the first 57
requests, and 80% can be identified in only 20 requests, with a
maximum false positive rate of 2.4%. In the time that this
system has been deployed on CoDeeN, robot-related abuse complaints
have dropped by a factor of 10.
- View the full text of this paper in HTML and PDF. Listen to the presentation in MP3 format.
Until June 2007, you will need your USENIX membership identification in order to access the full papers. The Proceedings are published as a collective work, © 2006 by the USENIX Association. All Rights Reserved. Rights to individual papers remain with the author or the author's employer. Permission is granted for the noncommercial reproduction of the complete work for educational or research purposes. USENIX acknowledges all trademarks within this paper.
- If you need the latest Adobe Acrobat Reader, you can download it from Adobe's site.
|
