A general framework for scalability and performance analysis of DHT routing systems

A general framework for scalability and performance analysis of DHT routing systems
of 11
All materials on our website are shared by users. If you have any questions about copyright issues, please report us to resolve them. We are always happy to assist you.
Related Documents
    a  r   X   i  v  :  c  s   /   0   6   0   3   1   1   2  v   1   [  c  s .   D   C   ]   2   8   M  a  r   2   0   0   6 A General Framework for Scalability and Performance Analysis of DHTRouting Systems Joseph S. Kong, Jesse S. A. Bridgewater and Vwani P. RoychowdhuryDepartment of Electrical EngineeringUniversity of California, Los Angeles {  jskong, jsab, vwani } Abstract  In recent years, many DHT-based P2P systems havebeen proposed, analyzed, and certain deployments havereached a global scale with nearly one million nodes. Oneis thus facedwith the questionof whichparticular DHT sys-tem to choose, and whether some are inherently more ro-bust and scalable. Toward developing such a comparative framework, we present the reachable component method (RCM) for analyzing the performance of different DHT routing systems subject to  random failures . We apply RCM to five DHT systems and obtain analytical expressions that characterize their   routability  as a continuous function of system size and node failure probability. An important con-sequenceis that in the large-networklimit, the routability of certainDHT systems goto zero for   any non-zeroprobabilityofnodefailure. TheseDHT routingalgorithmsaretherefore unscalable  , while some others, including Kademlia, which powers the popular eDonkey P2P system, are found to be scalable . 1 Introduction Developing scalable and fault tolerant systems to lever-age and utilize the shared resources of distributed comput-ers has been an important research topic since the dawn of computer networking. In recent years, the popularity andwidedeploymentofpeer-to-peer(P2P)systemshasinspiredthe development of distributed hash tables (DHTs). DHTstypically offer scalable  O (log n )  routing latency and effi-cient lookup interface. According to a recent study [12],the DHT based file-sharing network eDonkey is emergingas one of the largest P2P systems with millions of usersand accounting for the largest fraction of P2P traffic, whileP2P traffic currently accounts for 60% of the total Internetbandwidth. Given the transient nature of P2P users, analyz-ing and understanding the robustness of DHT routing algo-rithms in the asymptotic system size limit under unreliableenvironments become essential.In the past few years, there has been a growing numberof newly proposed DHT routing algorithms. However, inthe DHT routing literature, there have been few papers thatprovide a general analytical framework to compare acrossthe myriad routing algorithms. In this paper, we develop amethodto analyzethe performanceandscalability of differ-ent DHT routing systems under random failures of nodes.We would like to emphasize that we intend to analyze theperformance of the  basic  routing geometry and protocol.In a real system implementation, there is no doubt that asystem designer have many optional features, such as addi-tional sequential neighbors, to provide improved fault tol-erance. Nevertheless, the analysis of the basic routing ge-ometry will give us more insights and good guidelines tocompare among systems.In this paper, we investigate the routing performance of five DHT systems with uniform node failure probability  q  .Such a failure model, also known as the  static resilience model 1 , is assumed in the simulation study done by Gum-madi et al. [2]. A static failure model is well suited foranalyzing performance in the shorter time scale. In a DHT,very fast detection of faults is generally possible throughmeans such as TCP timeouts or keep-alive messages, butestablishing new connections to replace the faulty nodes ismoretime andresourceconsuming. The applicabilityof theresults derivedfrom this static model to dynamic situations,such as churn, is currently under study.Intuitively,asthenodefailureprobability q  increases,therouting performanceof the system will worsen. A quantita-tive metric, called  routability  is needed to characterize therouting performance of a DHT system under random fail-ure: Definition 1  The  routability  of a DHT routing system isthe expected number of routable pairs of nodes divided by 1 The term  static  refers to the assumption that a node’s routing tableremains unchanged after accounting for neighbor failures. 1  the expected number of possible pairs among the survivingnodes. In other words, it is the fraction of survived routing paths in the system. In general, routability is a function of the node failure probability  q   and system size  N  . As the DHT-basedeDonkeyis reachingglobal scale, it isimportant to study how DHT systems perform as the num-ber of nodes reaches millions or even billions. In fact, weknow from site percolationtheory [15], that if   q >  (1 −  p c ) ,where  p c  is called the percolation threshold of the underly-ing network, then the network will get fragmentedinto verysmall-size connected components and for large enough net-work size. As a result, the routability of the network willapproach zero for such failure probability due to the lack of connectivity. However, because of how messages getrouted as specified by the underlying routing protocol, allpairs belonging to the same connected component need notbe reachable under failure.In general, the size of the connected components do notdirectly give us the routability of the subnetworks. Hence,one needs to develop a framework different from the well-known framework of percolation. As a result, this work in-vestigates DHT routability under the random failure modelfor both finite system sizes and the infinite limit. We willdefine the  scalability  of a routing system as follows: Definition 2  A DHT routing system is said to be  scalable  if andonlyifits routabilityconvergestoanonzerovalueasthesystem size goes to infinity for a nonzero failure probability q  . Mathematically, it is defined as follows: lim N  →∞ r ( N,q  )  >  0  for  0  < q <  1 −  p c where  r ( N,q  )  denotes the routability of the system as a function of system size  N   and failure probability  q  . Simi-larly, the system is said to be  unscalable  if and only if itsroutability converges to zero as the system size goes to in- finity for a nonzero failure probability  q  : lim N  →∞ r ( N,q  ) = 0  for  0  < q <  1 −  p c We want to emphasize that in a real implementation, thereare many system parameters that the system designer canspecify, such as the number of near neighbors or sequentialneighbors. As a result, the designer can always add enoughsequential neighbors to achieve an acceptable routabilityunder reasonable node failure probability for a maximumnetworksizethatexceedstheexpectednumberofnodesthatwill participate in the system. The scalability definition isprovidedfor examining the  theoretical  asymptotic behaviorof DHT routing systems, not for claiming a DHT system isunsuitable for any large-scale deployment.Having specified the definition of the key metrics, wewill present the reachable component method (RCM), asimple yet effective method for analyzing DHT routing per-formanceunderrandomfailure. We applythe RCM methodtoanalyzethebasicroutingalgorithmsusedinthefollowingfive DHT systems: Symphony [10], Kademlia [11], Chord[16],CAN [14]andPlaxtonroutingbasedsystems[13]. Forall algorithms except Chord routing, we derive the analyt-ical expression for each algorithm’s routability under ran-dom failure, while an analytical expressionfor a tight lowerbound is obtained for Chord routing. In fact, our analyti-cal results match the simulation results carried out in [2],where different DHT systems were simulated and the per-centage of failed paths (i.e., 1-routability)was estimated for N   = 2 16 , as illustrated in Fig. 6. In addition,we also derivethe asymptotic performance of the routing algorithm underfailure as the system scales.One interesting finding of this paper is that under ran-dom failure, the basic DHT routing systems can be classi-fiedintotwoclasses:  scalable and unscalable . Forexample,the XOR routing scheme of Kademlia is found to be  scal-able ,sincetheroutabilityofthesystemundernonzeroprob-ability of failure converges very fast to a positive limit evenas the size of the system tends to infinity. This is consistentwith the observation that the Kademlia-based popular P2Pnetwork eDonkey is able to scale to millions of nodes. Incontrast,as thesystemscales, theroutabilityofSymphony’srouting scheme is found to quickly convergeto zero for anyfailure probabilitygreater than zero. Thus, the basic routingsystem for Symphony is found to be  unscalable . However,as briefly discussed above in this section, a system designerfor Symphony can specify enough near neighbors to guar-anteeanacceptableroutabilityin thesystemfora maximumnetwork size and a reasonable failure probability  q  .The rest of this paper is organized as follows. In section2, we discuss previous work on the fault tolerance of P2Prouting systems. In section 3, we will give an overview of the DHT routing systems that we intend to analyze. In sec-tion 4, we present the  reachable component method   (RCM)and apply the RCM method on several DHT systems. Insection 5, we examine the scalability of DHT routing sys-tems. In section 6, we give our concluding remarks. 2 Related Work Thestudyofrobustnessinroutingnetworkshasgrowninthe past few years with researchers simulating failure con-ditions in DHT-based systems. Gummadi et al. [2] showedthroughsimulationresults that the routinggeometryof eachsystem has a large effect on the network’s static resilienceto random failures. In addition, there have been researchwork done in the area of analyzing and simulating dynamicfailure conditions (i.e. churn) in DHT systems [5,7,8].Theory work has been done to predict the performanceof DHT systems under a static failure model. The two2  main approaches thus far have been graph theoretic meth-ods [1, 6, 9] and Markov processes [17]. Most analyticalwork to date has dealt with one or two routingalgorithms towhich their respective methods are well-suited but have notprovided comparisons across a large fraction of the DHTalgorithms. Angel et al. [1] use percolation theory to placetight bounds on the critical failure probability that can sup-port efficient routing on both hypercube and  d -dimensionalmesh topologies. By efficient they mean that it is possibleto route between two nodes with time complexity on theorder of the network diameter. While this method predictsthe point at which the network becomes virtually unusable,it does not allow the detailed characterization of routabil-ity as a function of the failure probability. In contrast, thereachable component method (RCM) method exploits thegeometries of DHT routing networks and leads to simpleanalytical results that predict routing performance for arbi-trary network sizes and failure probabilities. 3 Overview of DHT Routing Protocols We will first review the five DHT routingalgorithms thatwe intend to analyze. An excellent discussion of the geo-metric interpretation of these routing algorithms (except forSymphony) is provided by Gummadi et al. [2] and we usethe same terms for the geometric interpretations of DHTrouting systems in this paper (e.g. hypercube and ring ge-ometry for CAN and Chord routing systems, respectively).By followingthe algorithmdescriptions in [2] as well as thedescriptionsin this section, one can constructMarkov chainmodels (e.g. Fig. 4) for the DHT routing algorithms. Theapplication of the Markov chain models will be discussedin section 4.1 and 4.2. In addition, we will use the notation of   phases  as usedin [3]: we say thatthe routingprocess has reachedphase  j  if the numeric distance (used in Chord and Symphony) or theXOR distance (used in Kademlia) from the currentmessageholder to the target is between  2 j and  2 j +1 . In addition,we will use binary strings as identifiers although any otherbase besides 2 can be used. Finally, for those systems thatrequire resolving node identifier bits  in order  , we use theconvention of correcting bits from left to right. 3.1 Tree (Plaxton) Each node in a tree-based routing geometry has  log N  neighbors,with the  i th neighbormatchingthe first  i − 1  bitsand differ on the  i th bit. When a source node  S  , wishesto route to a destination,  D , the routing can only be suc-cessful if one of the neighbors of   S   , denoted  Z  , shares aprefix with  D  and has the highest-order differing bit. Eachsuccessful step in the routing results in the highest-orderbitbeing corrected until no bits differ.The routingMarkovchain(Fig. 4(a)) for the tree geome-trycaneasilybegeneratedbyexaminingthepossiblefailureconditions during routing. At each step in the routing pro-cess, the neighbor that will correct the leftmost bit must bepresent in order for the message to be routed. Otherwise,the message is dropped and routing fails. 3.2 Hypercube (CAN) In the hypercube geometry, each node’s identifier is abinary string representing its position in the  d -dimensionalspace. The distance between nodes is simply the Hammingdistanceof the two addresses. Thenumberof possiblepathsthat can correct a bit is reduced by 1 with each successfulstep in the route. This fact makes the creation of the hyper-cube routing Markov chain (Fig. 4(b)) straightforward. 3.3 XOR (Kademlia) In XOR routing [11], the distance between two nodes isthenumericvalueoftheXORoftheirnodeidentifiers. Eachnode keeps  log( N  )  connections, with the  i th neighbor cho-sen uniformlyat randomfroman XOR distancein the rangeof   [2 d − i , 2 d − i +1 ]  away. Messages are delivered by routinggreedily in the XOR distance at each hop. Moreover, it is asimpleexercisetoshow thatchoosinga neighborat anXORdistance of   [2 d − i , 2 d − i +1 ]  away is equivalent to choosing aneighbor by matching the first (i-1) bits of one’s identifier,flipping the  i th bit, and choose random bits for the rest of the bits.Effectively,thisconstructionisequivalenttothePlaxton-tree routinggeometry. As a result, when there is no failures,the XOR routing protocol resolves node identifier bits fromleft to right as in the Plaxton-treegeometry. However,whenthe system experiencesnode failures, nodes have the optionto route messages to neighbors that resolve lower order bitswhentheneighborthatwouldresolvethehighestorderbitisnot available. Note that resolving lower order bits will alsomake progress in terms of decreasing the XOR distance todestination. Nonetheless, the progress made by resolvinglower order bits is not necessarily preserved in future hopsor phases (see Fig. 5(a)).Forexample,atthestartoftheroutingprocess,onephaseis advancediftheneighborcorrectingtheleftmostbitexists.Otherwise, the routing process can correct one of the lowerorder bits. However, if all of the neighbors that would re-solve bits have failed, the routing process fails. A Markovchain model for the routing process is illustrated in Fig.5(b). 3.4 Ring (Chord) In Chord [16], nodes are placed in numerical orderaround a ring. Each node with identifier  a  maintains3  log( N  ) connectionsorfingers,witheachfingeratadistance [2 d − i , 2 d − i +1 ]  away (the randomized version of Chord isdiscussed here). Routing can be done greedily on the ring.When the system experiences failure, each node will con-tinue to route a message to the neighbor closest to destina-tion (i.e. in a greedy manner). A Markov chain model forthe routing process is illustrated in Fig. 8(a). 3.5 Small-World (Symphony) Small-world routing networks in the  1 -dimensional casehavea ring-likeaddressspacewhereeachnodeis connectedto a constant number of its nearest neighbors and a constantnumber of shortcuts that have a  1 /d  distance distribution ( d is the ring-distance between the end-points of the shortcut).Each node maintains a constant number of neighbors anduses greedy routing. Due to the distance distribution it willtake an average of   O (log N  )  hops before routing halves thedistanceto a target node,thereforerequiring O (log N  )  suchphases to reach a target node for a total expected latency of  O (log 2 N  ) .When the system experiences node failures, some of theshortcuts will be unavailable and the route will have to take”suboptimal” hops. The small-world Markov chain modelis fundamentally different from the ones for XOR routing(Fig. 5(b)) and ring routing (Fig. 8(a)). A routing phase is completed if any of the node’s shortcuts connects to thedesired phase. This happens with probability  k s d  where  k s denotes the number of shortcuts that each node maintains.Alternatively,theroutingfailsifallofthenode’snearneigh-borandshortcutconnectionsfail, which happenswith prob-ability  q  k n + k s . If neither of the above happens then theroute takes a suboptimal hop, which happens with probabil-ity  1  −  k s d  −  q  k n + k s . 4 Reachable Component Method and its Ap-plications 4.1 Method Description We now describe the steps of the  reachable component method   (RCM) in calculating the routabilityof a DHT rout-ing system under random failure. Before we delve into thedescription, let us first clarify several concepts and nota-tions on DHT routing. First, we allow all DHTs to fullypopulate their identifier spaces (i.e. node identifier length d  = log b  N  ). Second, when a DHT is not in its perfecttopologicalstate, it canbe thecasethat apairofnodesareinthe same connected component but these two nodes cannotroutebetweeneachother. Thus,thereachablecomponentof node  i  is the set of nodes that node  i  can route to under thegivenroutingalgorithm. Note that the reachablecomponentof node  i  is a subset of the connected component contain-ing node  i . Third, we assume that no ”back-tracking” isallowed (i.e. when a node cannot forward a message fur-ther, the node is not allowed to return the message back tothe node from whom the message was received).RCM is fairly simple in concept and involvesthe follow-ing five steps:1. Pick a random node, node  i , from the system and de-note it as the  root node . Construct the root node’s rout-ing topology from the routing algorithm of the system(i.e. the topology by which the root node routes to allother nodes in the system).2. Obtain the distribution of the distances (in hops or inphases) between the root node and all other nodes (de-noted as  n ( h ) ); in other words, for each integer  h , cal-culate the numberof nodes at distance  h  hops from theroot node. Note that the meaning of   hops  or  phases will be clear from the context.3. Compute the probability of success,  p ( h,q  ) , for rout-ing to a node  h  hops away from the root node under auniform node failure probability, q  .4. Computetheexpectedsizeofthe reachablecomponent  from the root node by first calculating the expectednumber of reachable nodes at distance  h  hops away(which is simply given by  n ( h )  ∗  p ( h,q  ) ). Now, wesum over all possible number of hops to obtain the ex-pected size of the reachable component.5. By inspection, the expected number of routable pairsin the system is givenby summingall survivingnodes’expected reachable component sizes. Then, dividingthe expected number of routable pairs by the numberof possible node pairs among all surviving nodes pro-duces the routabilityof the system underuniformnodefailure probability  q  .The formula for computing the expected size of thereachable component,  E  [ S  i ] , described in step 4 is derivedas follows: E  [ S  i ] =  E  [ N   j =1 j  = i Y  j ] = N   j =1 j  = i E  [ Y  j ] = d  h =1 n ( h )  p ( h,q  ) where  Y  j  is Bernoulli random variable for denoting reach-ing node  j , and  d  is the node identifier length.Since nodes in the system are removed with probability q  , there are  (1  −  q  ) N   or  pN   nodes that survive on average.In step 5, the formula for calculating the routability,  r , of the system under uniform failure probability  q   is given as4  Figure 1.  Here weillustrate the reachablecomponent method us-ing an 8-nodes hyper-cube. Figure 2.  We select node 011to be the root of the routinggraph. The symmetry of thesystem means that each nodewill be the root of a routinggraph with identical structure. h n ( h )  Pr ( S  h ,S  h +1 )1  31   1 − q  3 2  32   1 − q  2 3  33   1 − q  Figure 3.  For illustration purpose, we examine how011 routes a message to 100. Note that three choicesexist for the first hop, 2 choices exist for the second hopand only one choice left for the last hop. For this exam-ple,  p ( h,q  )  is:  p (3 ,q  ) = (1 − q  3 )(1 − q  2 )(1 − q  ) . follows: r  =  M  rp M   p = E    pN   i =1 S  i  2  ⌊  pN  ⌋ )2  ≈  pN   i =1 E  [ S  i ]  pN  (  pN   − 1)=  E  [ S  ](  pN   − 1)  (1)where  M  rp  denotes the expected number of   routable pairs among surviving nodes, and  M   p  is the expected number of all  possible pairs  among surviving nodes. Note that the lastequality follows from the observation that DHTs investi-gated in this paper have symmetric nodes. Therefore, therouting topology of each node is statistically identical toeach other. Thus, all  S  i ’s are identically distributed for all i ’s:  E  [ S  ] =  E  [ S  i ] ∀ i . 4.2 Using the Hypercube Geometry as anExample A simple application of the RCM method is illustratedfor the CAN hypercubic routing system in Fig. 1-3. TheRCM steps involved are as follows: Step 1.  As reviewed in section 3, in a hypercube routinggeometry [14], the distance (in hops) between two nodes istheir Hamming distance. Routing is greedy by correctingbits in any order for each hop. Step 2.  Thus, for any random node i  in a hypercuberoutingsystem with identifier length of   d  bits, we have the follow-ing distance distribution:  n ( h ) =  dh  . The justification isimmediate: a node at  h  hops away has a Hamming distanceof   h  bits with node  i . Since there are  dh   ways to place the h  differing bits, there are  dh   nodes at distance  h  (see Fig.2). Step 3.  The routing process can be modeled as a discretetime Markov chain (Fig. 3 and 4(b)). The states  S  ′ i s  of the Markov chain correspond to the number of correctedbits. Note that there are only two absorbing states in theMarkov chain: the failure state  F   and the success state (i.e. S  h ). Thus, the probabilityof successfully routingto a targetnode at distance  h  hops away is given by the probability of transitioning from  S  0  to  S  h  in the Markov chain model:  p ( h,q  ) =  Pr ( S  0  → S  1  → ... → S  h )=  Pr ( S  0  → S  1 ) Pr ( S  1  → S  2 ) ... Pr ( S  h − 1  → S  h )= (1 − q  h )(1 − q  h − 1 ) ... (1 − q  )= h  m =1 (1 − q  m )  (2) Step 4.  Thus, the expected size of the reachable componentis given as: E  [ S  ] = d  h =1 n ( h )  p ( h,q  ) = d  h =1  dh   h  m =1 (1 − q  m ) Step 5.  Using Eq. 1, we obtain the analytical expressionforroutability: r  = d  h =1 n ( h )  p ( h,q  )(1 − q  )2 d − 1  (3) = d  h =1  dh   h  m =1 (1 − q  m )(1 − q  )2 d − 1  (4)5
Similar documents
View more...
Related Search
We Need Your Support
Thank you for visiting our website and your interest in our free products and services. We are nonprofit website to share and download documents. To the running of this website, we need your help to support us.

Thanks to everyone for your continued support.

No, Thanks