Uploaded image for project: 'ZooKeeper'
  1. ZooKeeper
  2. ZOOKEEPER-1355

Add zk.updateServerList(newServerList)

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.5.0
    • Component/s: c client, java client
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
      Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.

      It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.

      We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

        Attachments

        1. loadbalancing.pdf
          288 kB
          Alexander Shraer
        2. loadbalancing-more-details.pdf
          301 kB
          Alexander Shraer
        3. ZOOKEEPER=1355-ver3.patch
          24 kB
          Alexander Shraer
        4. ZOOKEEPER-1355-10-Oct.patch
          89 kB
          Alexander Shraer
        5. ZOOKEEPER-1355-12-Oct.patch
          120 kB
          Marshall McMullen
        6. ZOOKEEPER-1355-13-Nov.patch
          121 kB
          Marshall McMullen
        7. ZOOKEEPER-1355-13-Nov-ver2.patch
          121 kB
          Marshall McMullen
        8. ZOOKEEPER-1355-13-Oct.patch
          119 kB
          Marshall McMullen
        9. ZOOKEEPER-1355-6-Nov.patch
          120 kB
          Marshall McMullen
        10. ZOOKEEPER-1355-6-Nov-ver2.patch
          120 kB
          Marshall McMullen
        11. ZOOKEEPER-1355-ver10.patch
          113 kB
          Marshall McMullen
        12. ZOOKEEPER-1355-ver10-1.patch
          114 kB
          Marshall McMullen
        13. ZOOKEEPER-1355-ver10-2.patch
          114 kB
          Marshall McMullen
        14. ZOOKEEPER-1355-ver10-3.patch
          241 kB
          Marshall McMullen
        15. ZOOKEEPER-1355-ver10-4.patch
          121 kB
          Marshall McMullen
        16. ZOOKEEPER-1355-ver10-4.patch
          114 kB
          Marshall McMullen
        17. ZOOKEEPER-1355-ver11.patch
          121 kB
          Marshall McMullen
        18. ZOOKEEPER-1355-ver11-1.patch
          121 kB
          Marshall McMullen
        19. ZOOKEEPER-1355-ver12.patch
          121 kB
          Marshall McMullen
        20. ZOOKEEPER-1355-ver12-1.patch
          120 kB
          Marshall McMullen
        21. ZOOKEEPER-1355-ver12-2.patch
          120 kB
          Marshall McMullen
        22. ZOOKEEPER-1355-ver12-4.patch
          119 kB
          Marshall McMullen
        23. ZOOKEEPER-1355-ver13.patch
          119 kB
          Marshall McMullen
        24. ZOOKEEPER-1355-ver14.patch
          119 kB
          Marshall McMullen
        25. ZOOKEEPER-1355-ver2.patch
          22 kB
          Alexander Shraer
        26. ZOOKEEPER-1355-ver4.patch
          27 kB
          Alexander Shraer
        27. ZOOKEEPER-1355-ver5.patch
          27 kB
          Alexander Shraer
        28. ZOOKEEPER-1355-ver6.patch
          32 kB
          Alexander Shraer
        29. ZOOKEEPER-1355-ver7.patch
          32 kB
          Alexander Shraer
        30. ZOOKEEPER-1355-ver8.patch
          33 kB
          Alexander Shraer
        31. ZOOKEEPER-1355-ver9.patch
          34 kB
          Alexander Shraer
        32. ZOOKEEPER-1355-ver9-1.patch
          34 kB
          Alexander Shraer
        33. ZOOOKEEPER-1355.patch
          22 kB
          Alexander Shraer
        34. ZOOOKEEPER-1355-test.patch
          24 kB
          Alexander Shraer
        35. ZOOOKEEPER-1355-ver1.patch
          22 kB
          Alexander Shraer

          Issue Links

            Activity

              People

              • Assignee:
                shralex Alexander Shraer
                Reporter:
                shralex Alexander Shraer
              • Votes:
                3 Vote for this issue
                Watchers:
                12 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: