Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8335

FSNamesystem should construct FSPermissionChecker only if permission is enabled

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.5.0, 2.6.0, 2.7.0, 2.8.0, 3.0.0-alpha1
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Only check permissions when permissions enabled in FSDirStatAndListingOp.getFileInfo() and getListingInt()

      Description

      FSNamesystem (2.5.x)/FSDirStatAndListingOp(current trunk) getFileInfo and getListingInt methods call getPermissionChecker() to construct a FSPermissionChecker regardless of isPermissionEnabled(). When permission checking is disabled, this leads to an unnecessary performance hit constructing a UserGroupInformation object that is never used.

      For example, from a stack dump when driving concurrent requests, they all end up blocking.

      Here's the thread holding the lock:

      "IPC Server handler 9 on 9000" daemon prio=10 tid=0x00007f78d8b9e800 nid=0x142f3 runnable [0x00007f78c2ddc000]
      java.lang.Thread.State: RUNNABLE
      at java.io.FileInputStream.readBytes(Native Method)
      at java.io.FileInputStream.read(FileInputStream.java:272)
      at java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
      at java.io.BufferedInputStream.read(BufferedInputStream.java:334)

      • locked <0x00000007d9b105c0> (a java.lang.UNIXProcess$ProcessPipeInputStream)
        at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283)
        at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325)
        at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177)
      • locked <0x00000007d9b1a888> (a java.io.InputStreamReader)
        at java.io.InputStreamReader.read(InputStreamReader.java:184)
        at java.io.BufferedReader.fill(BufferedReader.java:154)
        at java.io.BufferedReader.read1(BufferedReader.java:205)
        at java.io.BufferedReader.read(BufferedReader.java:279)
      • locked <0x00000007d9b1a888> (a java.io.InputStreamReader)
        at org.apache.hadoop.util.Shell$ShellCommandExecutor.parseExecResult(Shell.java:715)
        at org.apache.hadoop.util.Shell.runCommand(Shell.java:524)
        at org.apache.hadoop.util.Shell.run(Shell.java:455)
        at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
        at org.apache.hadoop.util.Shell.execCommand(Shell.java:791)
        at org.apache.hadoop.util.Shell.execCommand(Shell.java:774)
        at org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getUnixGroups(ShellBasedUnixGroupsMapping.java:84)
        at org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getGroups(ShellBasedUnixGroupsMapping.java:52)
        at org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback.getGroups(JniBasedUnixGroupsMappingWithFallback.java:50)
        at org.apache.hadoop.security.Groups.getGroups(Groups.java:139)
        at org.apache.hadoop.security.UserGroupInformation.getGroupNames(UserGroupInformation.java:1474)
      • locked <0x00000007a6df75f8> (a org.apache.hadoop.security.UserGroupInformation)
        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.<init>(FSPermissionChecker.java:82)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getPermissionChecker(FSNamesystem.java:3534)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getListingInt(FSNamesystem.java:4489)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getListing(FSNamesystem.java:4478)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getListing(NameNodeRpcServer.java:898)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getListing(ClientNamenodeProtocolServerSideTranslatorPB.java:602)
        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

      Here is (one of the many) threads waiting on the lock:

      "IPC Server handler 2 on 9000" daemon prio=10 tid=0x00007f78d8c48800 nid=0x142ec waiting for monitor entry [0x00007f78c34e3000]
      java.lang.Thread.State: BLOCKED (on object monitor)
      at org.apache.hadoop.security.UserGroupInformation.getGroupNames(UserGroupInformation.java:1472)

      • waiting to lock <0x00000007a6df75f8> (a org.apache.hadoop.security.UserGroupInformation)
        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.<init>(FSPermissionChecker.java:82)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getPermissionChecker(FSNamesystem.java:3534)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getListingInt(FSNamesystem.java:4489)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getListing(FSNamesystem.java:4478)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getListing(NameNodeRpcServer.java:898)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getListing(ClientNamenodeProtocolServerSideTranslatorPB.java:602)
        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

        Attachments

        1. HDFS-8335.3.patch
          3 kB
          Gabor Liptak
        2. HDFS-8335.2.patch
          3 kB
          Gabor Liptak
        3. HDFS-8335.patch
          3 kB
          Gabor Liptak

          Activity

            People

            • Assignee:
              gliptak Gabor Liptak
              Reporter:
              dmbryson David Bryson
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: