Each CoDNS node gathers and manages a set of neighbor nodes within a reasonable latency boundary. When a CoDNS instance starts, it sends a heartbeat to each node in the preconfigured CoDNS node list every second. The response contains the round trip time (RTT) and the average response time of the local nameserver at the peer node, reflecting the proximity and the availability of the peer node's nameserver. The top 10 nodes with different nameservers are picked as neighbors by comparing the sum with all nodes. Liveness of the chosen neighbors is periodically checked to see if the service is still available. One heartbeat is sent each second, so we guarantee the availability in 10 second granularity. Dead nodes are replaced with the next best node in the list.
Among these neighbor nodes, one peer is chosen for each remote name lookup using the Highest Random Weight (HRW) hashing scheme [22]. HRW consists of hashing the node name with the lookup name, and then choosing the node name with the smallest resulting hash value. Because HRW consistently picks the same node for the same domain name, this process enhances request locality for remote queries. Another desirable property of this approach is that some request locality is preserved as long as neighbor sets have some overlap. Full overlap is not required.
The number of neighbors is configurable according to the distribution of nodes. In the future, we will make CoDNS dynamically find the peer nodes not depending on the preconfigured set of nodes. One possible solution is to make each CoDNS node advertise its neighbor set and have a few well known nodes. Then, a new CoDNS node with no information about available CoDNS peer nodes can ask the well known nodes for their peer nodes and recursively gather the nodes by asking each neighbor until it finds a reasonable pool of CoDNS nodes.
Note that our neighbor discovery mechanisms are essentially advisory in nature - once the node has enough peers, it only needs to poll other nodes in order to have a reasonable set of candidates in case one of its existing peers becomes unavailable. In the event that some sites have enough peers to make this polling a scalability issue, each node can choose to poll a nearby subset of all possible peers to reduce the background traffic.