Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-7597

DelegationTokenIdentifier should cache the TokenIdentifier to UGI mapping

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: webhdfs
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Webhdfs seeks involve closing the current connection, and reissuing a new open request with the new offset. The RPC layer caches connections so the DN keeps a lingering connection open to the NN. Connection caching is in part based on UGI. Although the client used the same token for the new offset request, the UGI is different which forces the DN to open another unnecessary connection to the NN.

      A job that performs many seeks will easily crash the NN due to fd exhaustion.

        Attachments

        1. HDFS-7597.01.patch
          7 kB
          Xiao Chen
        2. HDFS-7597.patch
          5 kB
          Bob Hansen
        3. HDFS-7597.patch
          5 kB
          Daryn Sharp
        4. HDFS-7597.patch
          4 kB
          Daryn Sharp

          Issue Links

            Activity

              People

              • Assignee:
                daryn Daryn Sharp
                Reporter:
                daryn Daryn Sharp
              • Votes:
                0 Vote for this issue
                Watchers:
                23 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: