Trapeze messages are short control messages (maximum 128 bytes) with optional attached payloads typically containing application data not interpreted by the message system, e.g., a file block, a virtual memory page, or a TCP segment. Each message can have at most one payload attached to it. Separation of control messages and bulk data transfer is common to a large number of messaging systems since the V system [6].
A Trapeze control message and its payload (if any) are sent as a single packet on the network. Since Myrinet has no fixed maximum packet size (MTU), the maximum payload size of a Trapeze network is configurable, and is typically set to the virtual memory page size (4K or 8K). The Trapeze MTU is the maximum control message size plus the payload size.
Payloads are sent and received using DMA to/from aligned buffers residing anywhere in host memory. The host attaches a payload to an outgoing message using a Trapeze macro that stores the payload's DMA address and length into designated fields of the send ring entry. On the receiving side, Trapeze deposits the payload into a host memory buffer before delivering the control message.
The data structures in NIC memory include an endpoint structure shared with the host. A Trapeze endpoint (shown in Figure 2) includes two message rings, one for sending and one for receiving. Each message ring is a circular array of 128-byte control message buffers and related state, managed as a producer/consumer queue. From the perspective of a host CPU, the NIC produces incoming messages in the receive ring and consumes outgoing messages in the send ring. The host sends a message by forming it in the next free send ring entry and setting a bit to indicate that the message is ready to send. When a message arrives from the network, the firmware deposits it into the next free receive ring entry, sets a bit to inform the host that the message is ready to consume, and optionally signals the host with an interrupt.
Handling of incoming messages is interrupt-driven when Trapeze is used from within the kernel. Each kernel protocol module using Trapeze (i.e., gms_net and the IP network driver) registers a receiver interrupt handler upcalled from the Trapeze interrupt handler.
Trapeze is designed to optimize handling of payloads as well as to
deliver good performance for small messages. In a network memory
system, page fault stall time is determined primarily by the time to
transfer the requested page on the network. On the other hand, bursts
of page transfers (e.g., for read-ahead for sequential access) require
high bandwidth. The Trapeze firmware employs a message pipelining
technique called cut-through
delivery [27] to balance low payload latency with
high bandwidth under load. With this technique, the one-way raw
Trapeze latency for a 4K page transfer is 70
s on 300MHz
Pentium-II/440LX systems with LANai 4.1 M2M-PCI32 Myrinet adapters.
On these systems, Trapeze delivers 112 MB/s for a stream of 8K
payloads; with 64K payloads, Trapeze can use over 95% of the peak
bandwidth of the I/O bus, achieving 126 MB/s of user-to-user
point-to-point bandwidth.2
