Verification Phase.

This phase aims to verify whether a candidate URL from the pre-processing phase is malicious ( i.e., initiates a drive-by download). To do that, we developed a large scale web-honeynet that simultaneously runs a large number of Microsoft Windows images in virtual machines. Our system design draws on the experience from earlier work [25], and includes unique features that are specific to our goals. In what follows we discuss the details of the URL verification process.

Each honeypot instance runs an unpatched version of Internet Explorer. To inspect a candidate URL , the system first loads a clean Windows image then automatically starts the browser and instructs it to visit the candidate URL . We detect malicious URLs using a combination of execution based heuristics and results from anti-virus engines. Specifically, for each visited URL we run the virtual machine for approximately two minutes and monitor the system behavior for abnormal state changes including file system changes, newly created processes and changes to the system's registry. Additionally, we subject the HTTP responses to virus scans using multiple anti-virus engines. To detect malicious URLs , we develop scoring heuristics used to determines the likelihood that a URL is malicious. We determine a URL score based on a combined measure of the different state changes resulting from visiting the URL . Our heuristics score URLs based on the number of created processes, the number of observed registry changes and the number of file system changes resulting from visiting the URL .

To limit false positives, we choose a conservative decision criteria that uses an empirically derived threshold to mark a URL as malicious. This threshold is set such that it will be met if we detect changes in the system state, including the file system as well as creation of new processes. A visited URL is marked as malicious if it meets the threshold and one of the incoming HTTP responses is marked as malicious by at least one anti-virus scanner. Our extensive evaluation shows that this criteria introduces negligible false positives. Finally, a URL that meets the threshold requirement but has no incoming payload flagged by any of the anti-virus engines, is marked as suspicious.

On average, the detailed verification stage processes about one million URLs daily, of which roughly $25,000$ new URLs are flagged as malicious. The verification system records all the network interactions as well as the state changes. In what follows, we describe how we process the network traces associated with the detected malicious URLs to shed light on the malware distribution infrastructure.