ZooKeeper
  1. ZooKeeper
  2. ZOOKEEPER-1355

Add zk.updateServerList(newServerList)

    Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.5.0
    • Component/s: c client, java client
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
      Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.

      It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.

      We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

      1. loadbalancing.pdf
        288 kB
        Alexander Shraer
      2. loadbalancing-more-details.pdf
        301 kB
        Alexander Shraer
      3. ZOOKEEPER=1355-ver3.patch
        24 kB
        Alexander Shraer
      4. ZOOKEEPER-1355-10-Oct.patch
        89 kB
        Alexander Shraer
      5. ZOOKEEPER-1355-12-Oct.patch
        120 kB
        Marshall McMullen
      6. ZOOKEEPER-1355-13-Nov.patch
        121 kB
        Marshall McMullen
      7. ZOOKEEPER-1355-13-Nov-ver2.patch
        121 kB
        Marshall McMullen
      8. ZOOKEEPER-1355-13-Oct.patch
        119 kB
        Marshall McMullen
      9. ZOOKEEPER-1355-6-Nov.patch
        120 kB
        Marshall McMullen
      10. ZOOKEEPER-1355-6-Nov-ver2.patch
        120 kB
        Marshall McMullen
      11. ZOOKEEPER-1355-ver10.patch
        113 kB
        Marshall McMullen
      12. ZOOKEEPER-1355-ver10-1.patch
        114 kB
        Marshall McMullen
      13. ZOOKEEPER-1355-ver10-2.patch
        114 kB
        Marshall McMullen
      14. ZOOKEEPER-1355-ver10-3.patch
        241 kB
        Marshall McMullen
      15. ZOOKEEPER-1355-ver10-4.patch
        121 kB
        Marshall McMullen
      16. ZOOKEEPER-1355-ver10-4.patch
        114 kB
        Marshall McMullen
      17. ZOOKEEPER-1355-ver11.patch
        121 kB
        Marshall McMullen
      18. ZOOKEEPER-1355-ver11-1.patch
        121 kB
        Marshall McMullen
      19. ZOOKEEPER-1355-ver12.patch
        121 kB
        Marshall McMullen
      20. ZOOKEEPER-1355-ver12-1.patch
        120 kB
        Marshall McMullen
      21. ZOOKEEPER-1355-ver12-2.patch
        120 kB
        Marshall McMullen
      22. ZOOKEEPER-1355-ver12-4.patch
        119 kB
        Marshall McMullen
      23. ZOOKEEPER-1355-ver13.patch
        119 kB
        Marshall McMullen
      24. ZOOKEEPER-1355-ver14.patch
        119 kB
        Marshall McMullen
      25. ZOOKEEPER-1355-ver2.patch
        22 kB
        Alexander Shraer
      26. ZOOKEEPER-1355-ver4.patch
        27 kB
        Alexander Shraer
      27. ZOOKEEPER-1355-ver5.patch
        27 kB
        Alexander Shraer
      28. ZOOKEEPER-1355-ver6.patch
        32 kB
        Alexander Shraer
      29. ZOOKEEPER-1355-ver7.patch
        32 kB
        Alexander Shraer
      30. ZOOKEEPER-1355-ver8.patch
        33 kB
        Alexander Shraer
      31. ZOOKEEPER-1355-ver9.patch
        34 kB
        Alexander Shraer
      32. ZOOKEEPER-1355-ver9-1.patch
        34 kB
        Alexander Shraer
      33. ZOOOKEEPER-1355.patch
        22 kB
        Alexander Shraer
      34. ZOOOKEEPER-1355-test.patch
        24 kB
        Alexander Shraer
      35. ZOOOKEEPER-1355-ver1.patch
        22 kB
        Alexander Shraer

        Issue Links

          Activity

          Gavin made changes -
          Link This issue is related to ZOOKEEPER-1683 [ ZOOKEEPER-1683 ]
          Gavin made changes -
          Link This issue is related to ZOOKEEPER-1683 [ ZOOKEEPER-1683 ]
          Gavin made changes -
          Link This issue is depended upon by ZOOKEEPER-1660 [ ZOOKEEPER-1660 ]
          Gavin made changes -
          Link This issue blocks ZOOKEEPER-1660 [ ZOOKEEPER-1660 ]
          Alexander Shraer made changes -
          Link This issue is related too ZOOKEEPER-1683 [ ZOOKEEPER-1683 ]
          Alexander Shraer made changes -
          Link This issue blocks ZOOKEEPER-1660 [ ZOOKEEPER-1660 ]
          Alexander Shraer made changes -
          Link This issue relates to ZOOKEEPER-762 [ ZOOKEEPER-762 ]
          Flavio Junqueira made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Marshall McMullen made changes -
          Attachment ZOOKEEPER-1355-13-Nov-ver2.patch [ 12553464 ]
          Marshall McMullen made changes -
          Attachment ZOOKEEPER-1355-13-Nov.patch [ 12553459 ]
          Marshall McMullen made changes -
          Attachment ZOOKEEPER-1355-6-Nov-ver2.patch [ 12552304 ]
          Marshall McMullen made changes -
          Attachment ZOOKEEPER-1355-6-Nov.patch [ 12552299 ]
          Marshall McMullen made changes -
          Attachment ZOOKEEPER-1355-13-Oct.patch [ 12549007 ]
          Marshall McMullen made changes -
          Attachment ZOOKEEPER-1355-12-Oct.patch [ 12548996 ]
          Alexander Shraer made changes -
          Attachment ZOOKEEPER-1355-10-Oct.patch [ 12548688 ]
          Alexander Shraer made changes -
          Attachment ZOOKEEPER-1355-10-Oct.patch [ 12548653 ]
          Alexander Shraer made changes -
          Attachment ZOOKEEPER-1355-10-Oct.patch [ 12548653 ]
          Marshall McMullen made changes -
          Attachment ZOOKEEPER-1355-ver14.patch [ 12531858 ]
          Marshall McMullen made changes -
          Attachment ZOOKEEPER-1355-ver13.patch [ 12527742 ]
          Marshall McMullen made changes -
          Attachment ZOOKEEPER-1355-ver12-4.patch [ 12527735 ]
          Marshall McMullen made changes -
          Attachment ZOOKEEPER-1355-ver12-2.patch [ 12527539 ]
          Marshall McMullen made changes -
          Attachment ZOOKEEPER-1355-ver12-2.patch [ 12527538 ]
          Marshall McMullen made changes -
          Attachment ZOOKEEPER-1355-ver12-2.patch [ 12527538 ]
          Marshall McMullen made changes -
          Attachment ZOOKEEPER-1355-ver12-1.patch [ 12527535 ]
          Marshall McMullen made changes -
          Attachment ZOOKEEPER-1355-ver12.patch [ 12527533 ]
          Alexander Shraer made changes -
          Component/s c client [ 12312380 ]
          Marshall McMullen made changes -
          Attachment ZOOKEEPER-1355-ver11-1.patch [ 12520766 ]
          Marshall McMullen made changes -
          Attachment ZOOKEEPER-1355-ver11-1.patch [ 12520765 ]
          Marshall McMullen made changes -
          Attachment ZOOKEEPER-1355-ver11-1.patch [ 12520765 ]
          Marshall McMullen made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Marshall McMullen made changes -
          Attachment ZOOKEEPER-1355-ver11.patch [ 12520208 ]
          Marshall McMullen made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Marshall McMullen made changes -
          Attachment ZOOKEEPER-1355-ver10-4.patch [ 12518269 ]
          Marshall McMullen made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Marshall McMullen made changes -
          Attachment ZOOKEEPER-1355-ver10-4.patch [ 12518053 ]
          Marshall McMullen made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Marshall McMullen made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Marshall McMullen made changes -
          Attachment ZOOKEEPER-1355-ver10-3.patch [ 12518052 ]
          Marshall McMullen made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Marshall McMullen made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Marshall McMullen made changes -
          Attachment ZOOKEEPER-1355-ver10-2.patch [ 12518043 ]
          Marshall McMullen made changes -
          Attachment ZOOKEEPER-1355-ver10-1.patch [ 12518041 ]
          Marshall McMullen made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Marshall McMullen made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Marshall McMullen made changes -
          Attachment ZOOKEEPER-1355-ver10.patch [ 12518035 ]
          Marshall McMullen made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Alexander Shraer made changes -
          Link This issue relates to ZOOKEEPER-107 [ ZOOKEEPER-107 ]
          Alexander Shraer made changes -
          Link This issue relates to ZOOKEEPER-390 [ ZOOKEEPER-390 ]
          Alexander Shraer made changes -
          Link This issue supercedes ZOOKEEPER-338 [ ZOOKEEPER-338 ]
          Alexander Shraer made changes -
          Attachment ZOOKEEPER-1355-ver9-1.patch [ 12516137 ]
          Alexander Shraer made changes -
          Attachment ZOOKEEPER-1355-ver9.patch [ 12516134 ]
          Alexander Shraer made changes -
          Attachment ZOOKEEPER-1355-ver8.patch [ 12512043 ]
          Alexander Shraer made changes -
          Attachment ZOOKEEPER-1355-ver7.patch [ 12512037 ]
          Alexander Shraer made changes -
          Attachment ZOOKEEPER-1355-ver6.patch [ 12512026 ]
          Benjamin Reed made changes -
          Hadoop Flags Reviewed [ 10343 ]
          Alexander Shraer made changes -
          Attachment ZOOKEEPER-1355-ver5.patch [ 12511612 ]
          Alexander Shraer made changes -
          Attachment ZOOKEEPER-1355-ver4.patch [ 12511608 ]
          Alexander Shraer made changes -
          Attachment loadbalancing-more-details.pdf [ 12510782 ]
          Alexander Shraer made changes -
          Attachment ZOOKEEPER=1355-ver3.patch [ 12510231 ]
          Alexander Shraer made changes -
          Attachment ZOOKEEPER-1355-ver2.patch [ 12510149 ]
          Alexander Shraer made changes -
          Attachment ZOOOKEEPER-1355-test.patch [ 12510121 ]
          Alexander Shraer made changes -
          Attachment ZOOOKEEPER-1355-ver1.patch [ 12510108 ]
          Alexander Shraer made changes -
          Attachment ZOOOKEEPER-1355.patch [ 12510089 ]
          Alexander Shraer made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Alexander Shraer made changes -
          Field Original Value New Value
          Attachment loadbalancing.pdf [ 12509967 ]
          Alexander Shraer created issue -

            People

            • Assignee:
              Alexander Shraer
              Reporter:
              Alexander Shraer
            • Votes:
              3 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development