Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-5010

Reduce the frequency of getCurrentUser() calls from namenode

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.1.0-beta, 0.23.9
    • Fix Version/s: 2.1.0-beta, 0.23.10
    • Component/s: namenode, performance
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      According to the "worst" jstack of a busy namenode I took, there were 29 BLOCKED handler threads. All of them were blocked on the UGI class lock. Here is the breakdown:

      • 2 ensureInitialized() - from non static synchronized methods. HADOOP-9748 will unblock these.
      • 27 getCurrentUser()

      Among the 27 threads that were blocked at getCurrentUser(),

      • 18 FSPermissionChecker() - from FSNamesystem#getPermissionChecker() in most namenode RPC methods
      • 8 BlockTokenSecretManager#generateToken() - getBlockLocations()
      • 1 NameNodeRpcServer.mkdirs

      I think FSPermissionChecker can be modified to be created with a passed in UGI. FSNamesystem can the one already stored in RPC server by calling getRemoteUser(). This will eliminate a bulk of getCurrentUser() calls from namenode RPC handlers. A similar change can be made to mkdirs. Block token generation is not as straightforward. Even without it we can eliminate majority of the calls.

        Attachments

        1. HDFS-5010.b023.patch
          8 kB
          Kihwal Lee
        2. HDFS-5010.b023.patch
          6 kB
          Kihwal Lee
        3. HDFS-5010.patch
          8 kB
          Kihwal Lee
        4. HDFS-5010.patch
          7 kB
          Kihwal Lee
        5. HDFS-5010.patch
          5 kB
          Kihwal Lee

          Issue Links

            Activity

              People

              • Assignee:
                kihwal Kihwal Lee
                Reporter:
                kihwal Kihwal Lee
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: