Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-16828

Zookeeper Delegation Token Manager fetch sequence number by batch

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 3.3.1, 3.4.0
    • None
    • None
    • Reviewed

    Description

      Currently in ZKDelegationTokenSecretManager.java the seq number is incremented by 1 each time there is a request for creating new token. This will need to send traffic to Zookeeper server. With multiple managers running, there is data contention going on. Also, since the current logic of incrementing is using tryAndSet which is optimistic concurrency control without locking. This data contention is having performance degradation when the secret manager are under volume of traffic.

      The change here is to fetching this seq number by batch instead of 1, which will reduce the traffic sent to ZK and make many operations inside ZK secret manager's memory.

      After putting this into production we saw huge improvement to the RPC processing latency of get delegationtoken calls. Also, since ZK takes less traffic in this way. Other write calls, like renew and cancel delegation tokens are benefiting from this change.

       

       

      Attachments

        1. Screen Shot 2020-01-25 at 2.25.16 PM.png
          61 kB
          Fengnan Li
        2. Screen Shot 2020-01-25 at 2.25.24 PM.png
          82 kB
          Fengnan Li
        3. Screen Shot 2020-01-25 at 2.25.06 PM.png
          79 kB
          Fengnan Li
        4. HADOOP-16828.001.patch
          8 kB
          Fengnan Li
        5. HADOOP-16828.002.patch
          9 kB
          Fengnan Li

        Issue Links

          Activity

            People

              fengnanli Fengnan Li
              fengnanli Fengnan Li
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: