Weiqing Sun, Varun Katta, Kumar Krishna and R. Sekar
Department of Computer Science
Stony Brook University, Stony Brook, NY 11794
Even if malware is deployed on a tightly controlled network, and prevented from accessing the Internet by a firewall, there is no guarantee that malware will not access (or escape into) the Internet. In particular, malware may discover and exploit vulnerabilities in the very same network components that are supposed to prevent it from accessing the Internet, such as VLAN switches, routers and firewalls. Alternatively, it may exploit vulnerabilities in network monitoring and intrusion detection applications such as Snort, Wireshark (formerly Ethereal), tcpdump or other network-exposed applications that may be run on the network used for security experiments. Numerous CVE vulnerabilities have been reported involving these applications' components in the past several years, and hence the likelihood of finding exploitable vulnerabilities is non-negligible. When combined with the large magnitude of damage that can result from malware escaping to the Internet, the overall risk may be unacceptable. These risks grow in proportion to the size of experimental testbeds -- for instance, an oft-voiced concern about security experiments in the context of the proposed GENI infrastructure is that it should not become the world's most powerful botnet!
A typical approach for mitigating the security concerns mentioned above is to physically disconnect experimental networks from the Internet, but this requires researchers to have physical access to these networks -- something that is not possible in the context of most large-scale testbeds. To mitigate this difficulty, DETER [1] (which is based on Emulab) can allow remote access to hosts used in security experiments, while mitigating risks by deploying firewalls, network access control mechanisms and/or IDS/IPS. However, errors in configuring these mechanisms may result in the possibility that malicious code may escape into the Internet. As a result, DETER recommends no remote access for the most dangerous experiments. But disabling remote access for a particular experiment is insufficient, it is possible that malicious code exploit vulnerabilities in the software/hardware components of the testbed infrastructure and escape to other experiments that have remote access.
To provide controlled Internet-access for carrying out these experiments, we propose a new approach that uses datalink layer virtualization in conjunction with virtualization of hosts involved in security experiments. It is implemented by rewriting every network packet created within a security experiment in such a way as to isolate these packets from the underlying network infrastructure. As a result, malicious packets cannot ``escape'' outside the experimentation network, and therefore have no opportunities to exploit vulnerabilities in servers, firewalls or routers within production environments. Remote access can be provided by creating a communication tunnel from an experimenter's workstation to the experimentation network.
The key insight behind our approach is that network virtualization can be used to provide security in much the same way as host virtualization. Just as a VMM mediates all accesses made by a guest OS to host hardware, our approach relies on a packet rewriter deployed on the host machine that mediates all accesses to the physical network made by a guest VM. We point out that while today's security technologies such as anti-virus and application firewalls can provide some defense against malicious code, they are not considered strong enough to run malware experiments. This is because malware can potentially compromise its operating system, and subsequently disable or defeat all defenses deployed on this OS. In contrast, since a VMM is a much simpler piece of software that provides a much narrower interface to a guest OS as compared to the complexity of an application-to-OS interface, it is believed to provide an adequate level of security for experiments involving malware. In the same manner, our packet rewriter is a small piece of code with very simple functionality, and hence can be relied upon to ensure isolation of the virtual networks from the underlying physical network.
Our packet rewriter encapsulates the guest VM's network packets in a
manner that they will no longer be interpreted by the underlying network
fabric. In particular, consider a datalink layer packet 
from a host
to another host 
, where and are part of a security
experiment. In V-NetLab, both and will be implemented as guests on
host machines 
and 
respectively. The packet is intercepted
by a packet rewriter (implemented using a kernel module) on host ,
which generates a new datalink layer packet 
with the source address
of , destination address , and a protocol identifier ETH_P_VNETLAB
that is unused in the (physical) testbed. The payload of is the
entire packet . On , the kernel hands packets with the protocol
identifier of ETH_P_VNETLAB to our packet rewriter, which inverts the
above transformation
and hands to the guest . Due to the fact that looks like any
other datalink layer packet from to , it is highly unlikely to
compromise any components on the physical testbed that operate at these
datalink layer. Moreover, since the protocol identifier of
ETH_P_VNETLAB is unknown to
these components, they are unlikely to inspect or process its payload. As
a result, components on the physical network are highly unlikely to be
compromised (or affected in any way) by network traffic generated as part
of security experiments. Additionally, the entire payload of can be
encrypted in order to ensure that its contents remain confidential, or to
ensure that the resulting payload looks essentially random (i.e.,
uncorrelated with the original packet contents) and hence cannot
predictably be used to exploit vulnerabilities on these devices/services.
A benefit of virtualization at the datalink layer is that it permits the use of any layer-3 protocol within the security experiments, including IP, ICMP, ARP, etc. Moreover, since the packets transmitted by guest OSes remain encapsulated on the physical network, it is possible for different virtual networks to use overlapping IP addresses without interfering with each other. Indeed, the management of IP addresses on the virtual network is totally under the control of the user of the virtual network; these addresses need not be allocated or approved by the operators of the underlying physical network. We remark that our mechanisms for remote connectivity operate independent of such overlaps in IP address space.
In the rest of this paper, we first describe the design and implementation of our approach in Section 2. Several issues about our approach are discussed in Section 3. Related work is discussed in Section 4, followed by the concluding remarks in Section 5.
We use Linux as the host OS to simplify development of virtual network control and management software. The virtual machines (guest OSes) themselves may run Linux, Windows, or any other OS supported by the virtual machine software. Our implementation uses VMware for virtualization, since it provides seamless support for different guest OSes, strong isolation, and reasonably good performance.
We have implemented V-NetLab framework on a hardware platform consisting of dual-processor workstations connected together by a switched gigabit network. Also included in the platform is an NFS server with a large enough disk to accommodate many virtual machine images. Since our platform is relatively small (currently about 10 workstations), our network infrastructure consists of a gigabit Ethernet switch. With this physical infrastructure, we have been able to support around 50 experiments, each consisting of up to ten hosts. These experiments were carried out by students in the context of network and system security courses, and involved a variety of tasks including network mapping and sniffing, network intrusion detection, experiments with malicious code, exploit development, etc. 1
We note that scaling to a larger testbed is primarily a resource management and mapping problem, i.e., it requires the development of algorithms and techniques to map logical networks onto the physical hosts. Development and implementation of these techniques is ongoing effort. At present, we assume that this mapping has been generated and given to the V-NetLab runtime system, which is responsible for managing the physical infrastructure and interacting with remote users. This runtime system provides management commands that may be used by V-NetLab users to create virtual networks and instantiate them, query their status, etc. The components of V-NetLab runtime system are shown in Figure 1.
A virtual network typically consists of a number of virtual machines distributed over a set of physical hosts. Virtual machines belonging to the same subnet may be distributed over multiple physical hosts in the testbed. In that case, our packet handler is responsible for transparent transmission of packets among virtual machines via physical hosts. In order to achieve this, the packet handlers maintain the mapping between the virtual network topology (also called logical topology) and the physical topology. We use two data structures to maintain the mapping information. Given a logical topology, logical interfaces that are directly connected together by one or more hubs form a sibling closure (SC). A logical host interface in an SC should be able to listen to network traffic originating at (or destined to) any of the other logical host interfaces in the same SC. The SCs may then be connected together using (logical) switches. Therefore, the logical topology provided by the user can be divided into multiple SCs.
Another important data structure is Participation Table or PT, which lists the set of vmnet interfaces that belong to an SC. This is generated during the start-up phase of the virtual network. By definition, each vmnet interface in an SC is able to listen to conversations involving any other vmnet interface of that SC. The packet rewriter is responsible for ensuring this by relaying packets across the vmnet interfaces of different hosts that are part of the same SC. Figure 2 illustrates these concepts. To reduce clutter, only a single vmnet interface is shown for each host, but in reality, each physical host can support about 100 vmnet interfaces. In this figure, vm2, vm3 and vm4 are supposed to be on the same logical hub. Since vm2 and vm3 have been mapped to the same physical host, they can share a single vmnet interface, called vmnet1 in the figure. vm4 needs a separate vmnet interface on Host 2, called vmnet3. The interfaces vmnet1 and vmnet3 are participants in an SC. Thus, our packet rewriter will ensure that any packet from any of the hosts vm2, vm3 and vm4 would be relayed on both vmnet1 and vmnet3. Virtual network packets observed by the packet rewriter that do not originate from the participants in the SC will be dropped by the rewriter to ensure isolation. An exception to this rule is made for packets originating on the respective physical host. For instance, Host 1 can transmit packets to vm2 or vm3 -- the packet rewriter will handle them appropriately. This ability plays an important role in providing external tunnels into virtual networks.
Virtual network packets that need to be forwarded on a remote vmnet interface have to be transmitted on physical wire through host physical interface (e.g., eth0). To transparently transmit packets generated by a virtual machine on to the physical wire, we need a mechanism to grab every packet arriving at the vmnet interface and forward it on to the physical host's network interface (eth0) as specified in PT. Moreover, the packet needs to be captured at eth0 interface of the receiving physical host and then forwarded to participating vmnet interfaces on that physical host. These are ensured by our packet rewriter, which is implemented as an extra layer of packet handler (using dev_add_pack in Linux) to intercept all Ethernet packets, and then encapsulate the packet using a special data link layer protocol ETH_P_VNETLAB with the source/destination MAC addresses as the MAC address of the sending host network interface and receiving host interface2 . On the receiving host, the packet handler simply removes the encapsulated header and forwards the packet to corresponding vmnet interfaces according to the PT.
Note that, in order to access V-NetLab, a user is required to
first establish an SSH session with a gateway machine to the testbed.
Hence our approach relies on using this connection to tunnel network
packets to the target virtual network. A natural way to do this
is to use the tunneling capability provided by ssh, which allows
data targeted at a specified TCP port on the ssh client machine to
be forwarded so as to reach a destination port on the
server machine and vice-versa. Using this capability, a tunnel
is created from port on the user's workstation to port
on the gateway. Next, another SSH tunnel is created from port
on the gateway to port 
on the physical host that hosts the target
virtual machine. Finally, Linux iptables is used to forward packets
from port to port 22 (i.e., the SSH port) of the target virtual
machine. Figure 3 illustrates this process. Our management
infrastructure automatically picks suitable values for ports , ,
and , and sets up the tunnels. It also adds additional iptables
rules on the gateway to ensure that a user can only send packets
to the ports corresponding to the tunnels created by her. This ensures
that one user cannot (accidentally or intentionally) obtain network
connectivity with the virtual networks belonging to other users.
Virtual networks realized using our approach are isolated from each other, and from physical network infrastructure. A mechanism for controlled external connectivity into this network is provided, but this mechanism cannot be used by a malicious virtual machine to send data to arbitrary external hosts; instead, data may be sent only to a single host and port number to which a tunnel has been explicitly set up.
Another way to think about our network virtualization approach is that it enforces a strong form of ``default deny'' policy: network packets belonging to a virtual network cannot be seen by any host (virtual or physical) unless explicitly permitted through the deliberate process of network virtualization and tunnel creation.
We point out that security based on isolation provided by virtual machine software can become weakened if malware running inside the virtual machine can exploit vulnerabilities in virtual machine software (e.g., hypervisors) so as to attack the host environment. As far as we know, such attacks have not become a real-world threat. Moreover, since virtual machine software is simpler and provides a narrower interface to a guest OS as compared to the application-to-OS interface, it is believed to be feasible to harden this layer.
Another target for malware attacks is the packet rewriter, which has to operate on packets that may be created by malware. However, due to the small size and conceptual simplicity of this module, we believe that the likelihood of finding exploitable vulnerabilities in this module is rather small.
As a testbed for security experiments, V-NetLab might become an attractive target for attacks. We rely on best security practices to mitigate this threat, such as the installation of a minimal set of applications and services on the gateway machine, providing only a restricted shell for users that permits them to invoke only the commands relevant for managing their virtual networks, etc.
In order to enhance usability, V-NetLab provides a controlled remote access functionality, which can potentially be used to transport malicious code or data outside of the virtual networks, or perhaps to carry out an attack on external hosts. Our mechanisms are designed so that this is possible only with active involvement by the user of a virtual network. They have to explicitly set up these tunnels; and even then, connectivity is provided only to their desktops. If malicious code/data needs to reach elsewhere, the user needs to propagate the data further from his workstation, which requires additional deliberate actions on his behalf. In other words, if the user of a virtual network is malicious, then such outflow of malicious data/code can easily occur. Otherwise, we believe that the risk is very small. For self-spreading worms, they would need propagation channels such as email, IRC and instant messaging. The controlled tunnel created in V-NetLab is specific to a particular application, e.g., ssh. And only the ssh client on the user's desktop is exposed to the virtual network. Therefore, the likelihood of being used as an automatic worm spreading method is greatly minimized.
Our V-NetLab is currently LAN-centric. This does not mean that it cannot support complex topologies, but it certainly means that our focus has been on relatively small networks. In fact, it can support networks that involve multiple hops between senders and receivers by assigning dedicated virtual machines as virtual routers.
Although approaches such as [7,10,6] have simplified the setup of individual hosts for security experiments, creation of entire networks is still cumbersome. Planetlab [3] is a distributed laboratory that provides convenient management tools to control a large collection of hosts that run identical software. Emulab [2] is another similar approach that provides such facilities. Emulab supports light-weight virtualization, based on FreeBSD Jails. But both approaches do not provide support for running security experiments, where damage can escape to the Internet. Another drawback is that they do not provide the degree of flexibility needed for our approach, where computers running different OSes may need to be hosted on the same physical machine. An alternative mode supported in Emulab is one where physical nodes on the testbed can be dedicated to run a custom OS image. This approach provides the desired degree of flexibility to support security experiments, but does not allow sharing of underlying hardware across multiple OSes.
DETER [1] is a project which aims at building a testbed for security research based on Emulab. In addition to isolating different experiments using VLANs, firewall and intrusion detection system are deployed to prevent potentially malicious experiments from affecting the Internet. However, as mentioned earlier, there exists a possibility that malware may be able to subvert these defenses by exploiting vulnerabilities in network-exposed hardware and software. In contrast, we believe that V-NetLab provides stronger isolation guarantees, while still providing safe external connectivity.
VNET [8] and VIOLIN [4] have some similarity with our work in terms of the support for virtual networks. VNET is concerned with distributed computing applications, and their virtual networks span a wide-area network. Their approach is based on tunneling Ethernet packets over TCP/IP. VIOLIN uses an application-level virtual network architecture built on top of an overlay infrastructure such as Planetlab. They use UDP tunneling in the Internet domain to emulate the physical layer in the VIOLIN domain.
Unlike VNET and VIOLIN, our approach achieves network virtualization at the datalink layer. This provides better performance, as it eliminates the need for higher layers of the protocol from having to process the same packet twice. Moreover, our approach supports multiple virtual networks to share the same set of IP addresses, thus make the management task easier. On the other hand, our approach currently does not support experiments that span across the Internet.
Wroclawski et al [9] propose a collaborative specification of constraints from both experimenters and testbed operators in order to achieve the balance of both security and usability. Our approach does not make any assumption that experimenters provide constraint requirement of their experiments, and can provide satisfactory security and usability even in face of malware experimentation. On the other hand, our approach can potentially achieve better usability (e.g., controlled tunnel creation can be made less constrained.) with accurate constraint information from the experimenters.
Krishna et al [5] presented our earlier version of V-NetLab with the focus on using it to support security course projects in a class setting. This paper focuses on our approach for realizing logically isolated networks for supporting security experiments.
In this paper, we presented our approach for realizing virtual networks for security experiments. The datalink layer virtualization approach provides strong isolation for virtual networks, while providing the flexibility for establishing connectivity to external networks in a controlled way.
We implemented this approach into V-NetLab framework and have so far used it in several security courses in our department to provide hands-on projects for students. Student response to date has been very positive.
![\includegraphics[width=3in, height=2.3in]{arch.eps}](img7.png)
![\includegraphics[width=3in, height=1.3in]{virtual_hub.eps}](img8.png)
![\includegraphics[width=3.2in, height=2in]{remote_ssh.eps}](img9.png)