Uploaded image for project: 'ZooKeeper'
  1. ZooKeeper
  2. ZOOKEEPER-2849

Quorum port binding needs exponential back-off retry

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 3.4.6, 3.5.3
    • Fix Version/s: None
    • Component/s: quorum
    • Labels:
      None

      Description

      Recently we upgraded the AWS instance type we use for running out ZooKeeper nodes, and by doing so we're intermittently hitting an issue where ZooKeeper cannot bind to the server election port because the IP is incorrect. This is due to name resolution in Route53 not being in sync when ZooKeeper starts on the more powerful EC2 instances. Currently in QuorumCnxManager.Listener, we only attempt to bind 3 times with a 1s sleep between retries, which is not long enough.

      I'm proposing to change this to follow an exponential back-off type strategy where each failed attempt causes a longer sleep between retry attempts. This would allow for Zookeeper to gracefully recover when the host is misconfigured, and subsequently corrected, without requiring the process to be restarted while also minimizing the impact to the running instance.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                brian.lininger@gmail.com Brian Lininger
                Reporter:
                brian.lininger@gmail.com Brian Lininger
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated: