Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-5420

Check If HDFS ACLs are enabled before trying to get the ACLs

    Details

    • Epic Color:
      ghx-label-5

      Description

      While Impala is loading block information a lot of HDFS exceptions are printed to the Catalog log, Impala swallows those exceptions and metadata loading completes fine.

      Yet these exceptions slow down metadata loading significantly and cause quiet a bit of memory churn.

      There's an ACL bit that Impala can query on the FsPermission of the FileStatus to see if there is an ACL worth fetching:

      https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/permission/FsPermission.java#L303

      0531 23:09:45.532622 92906 RetryInvocationHandler.java:126] Exception while invoking class org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getAclStatus over null. Not retrying because try once and fail.
      Java exception follows:
      org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.AclException): The ACL operation has been rejected.  Support for ACLs has been disabled by setting dfs.namenode.acls.enabled to false.
      	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAclsConfigFlag(FSNamesystem.java:9637)
      	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAclStatus(FSNamesystem.java:9111)
      	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getAclStatus(NameNodeRpcServer.java:1620)
      	at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getAclStatus(AuthorizationProviderProxyClientProtocol.java:907)
      	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getAclStatus(ClientNamenodeProtocolServerSideTranslatorPB.java:1335)
      	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
      	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
      	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
      	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2217)
      	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2213)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:415)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
      	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2211)
      
      	at org.apache.hadoop.ipc.Client.call(Client.java:1502)
      	at org.apache.hadoop.ipc.Client.call(Client.java:1439)
      	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
      	at com.sun.proxy.$Proxy17.getAclStatus(Unknown Source)
      	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getAclStatus(ClientNamenodeProtocolTranslatorPB.java:1336)
      	at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:606)
      	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:258)
      	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
      	at com.sun.proxy.$Proxy18.getAclStatus(Unknown Source)
      	at org.apache.hadoop.hdfs.DFSClient.getAclStatus(DFSClient.java:3353)
      	at org.apache.hadoop.hdfs.DistributedFileSystem$42.doCall(DistributedFileSystem.java:2059)
      	at org.apache.hadoop.hdfs.DistributedFileSystem$42.doCall(DistributedFileSystem.java:2056)
      	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
      	at org.apache.hadoop.hdfs.DistributedFileSystem.getAclStatus(DistributedFileSystem.java:2056)
      	at org.apache.impala.util.FsPermissionChecker.getPermissions(FsPermissionChecker.java:288)
      	at org.apache.impala.catalog.HdfsTable.getAvailableAccessLevel(HdfsTable.java:787)
      	at org.apache.impala.catalog.HdfsTable.createPartition(HdfsTable.java:861)
      	at org.apache.impala.catalog.HdfsTable.loadAllPartitions(HdfsTable.java:639)
      	at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1065)
      	at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1002)
      	at org.apache.impala.catalog.TableLoader.load(TableLoader.java:80)
      	at org.apache.impala.catalog.TableLoadingMgr$2.call(TableLoadingMgr.java:237)
      	at org.apache.impala.catalog.TableLoadingMgr$2.call(TableLoadingMgr.java:234)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:745)
      W0531 23:09:45.538826 92906 RetryInvocationHandler.java:126] Exception while invoking class org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getAclStatus over null. Not retrying because try once and fail.
      Java exception follows:
      org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.AclException): The ACL operation has been rejected.  Support for ACLs has been disabled by setting dfs.namenode.acls.enabled to false.
      	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAclsConfigFlag(FSNamesystem.java:9637)
      	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAclStatus(FSNamesystem.java:9111)
      	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getAclStatus(NameNodeRpcServer.java:1620)
      	at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getAclStatus(AuthorizationProviderProxyClientProtocol.java:907)
      	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getAclStatus(ClientNamenodeProtocolServerSideTranslatorPB.java:1335)
      	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
      	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
      	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
      	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2217)
      	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2213)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:415)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
      	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2211)
      
      	at org.apache.hadoop.ipc.Client.call(Client.java:1502)
      	at org.apache.hadoop.ipc.Client.call(Client.java:1439)
      	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
      	at com.sun.proxy.$Proxy17.getAclStatus(Unknown Source)
      	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getAclStatus(ClientNamenodeProtocolTranslatorPB.java:1336)
      	at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:606)
      	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:258)
      	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
      	at com.sun.proxy.$Proxy18.getAclStatus(Unknown Source)
      	at org.apache.hadoop.hdfs.DFSClient.getAclStatus(DFSClient.java:3353)
      	at org.apache.hadoop.hdfs.DistributedFileSystem$42.doCall(DistributedFileSystem.java:2059)
      	at org.apache.hadoop.hdfs.DistributedFileSystem$42.doCall(DistributedFileSystem.java:2056)
      	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
      	at org.apache.hadoop.hdfs.DistributedFileSystem.getAclStatus(DistributedFileSystem.java:2056)
      	at org.apache.impala.util.FsPermissionChecker.getPermissions(FsPermissionChecker.java:288)
      	at org.apache.impala.catalog.HdfsTable.getAvailableAccessLevel(HdfsTable.java:787)
      	at org.apache.impala.catalog.HdfsTable.createPartition(HdfsTable.java:861)
      	at org.apache.impala.catalog.HdfsTable.loadAllPartitions(HdfsTable.java:639)
      	at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1065)
      	at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1002)
      	at org.apache.impala.catalog.TableLoader.load(TableLoader.java:80)
      	at org.apache.impala.catalog.TableLoadingMgr$2.call(TableLoadingMgr.java:237)
      	at org.apache.impala.catalog.TableLoadingMgr$2.call(TableLoadingMgr.java:234)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:745)
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                bharathv bharath v
                Reporter:
                mmokhtar Mostafa Mokhtar
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: