Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-5420

Check If HDFS ACLs are enabled before trying to get the ACLs

    XMLWordPrintableJSON

Details

    • ghx-label-5

    Description

      While Impala is loading block information a lot of HDFS exceptions are printed to the Catalog log, Impala swallows those exceptions and metadata loading completes fine.

      Yet these exceptions slow down metadata loading significantly and cause quiet a bit of memory churn.

      There's an ACL bit that Impala can query on the FsPermission of the FileStatus to see if there is an ACL worth fetching:

      https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/permission/FsPermission.java#L303

      0531 23:09:45.532622 92906 RetryInvocationHandler.java:126] Exception while invoking class org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getAclStatus over null. Not retrying because try once and fail.
Java exception follows:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.AclException): The ACL operation has been rejected.  Support for ACLs has been disabled by setting dfs.namenode.acls.enabled to false.
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAclsConfigFlag(FSNamesystem.java:9637)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAclStatus(FSNamesystem.java:9111)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getAclStatus(NameNodeRpcServer.java:1620)
	at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getAclStatus(AuthorizationProviderProxyClientProtocol.java:907)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getAclStatus(ClientNamenodeProtocolServerSideTranslatorPB.java:1335)
	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2217)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2213)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2211)

	at org.apache.hadoop.ipc.Client.call(Client.java:1502)
	at org.apache.hadoop.ipc.Client.call(Client.java:1439)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
	at com.sun.proxy.$Proxy17.getAclStatus(Unknown Source)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getAclStatus(ClientNamenodeProtocolTranslatorPB.java:1336)
	at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:258)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
	at com.sun.proxy.$Proxy18.getAclStatus(Unknown Source)
	at org.apache.hadoop.hdfs.DFSClient.getAclStatus(DFSClient.java:3353)
	at org.apache.hadoop.hdfs.DistributedFileSystem$42.doCall(DistributedFileSystem.java:2059)
	at org.apache.hadoop.hdfs.DistributedFileSystem$42.doCall(DistributedFileSystem.java:2056)
	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
	at org.apache.hadoop.hdfs.DistributedFileSystem.getAclStatus(DistributedFileSystem.java:2056)
	at org.apache.impala.util.FsPermissionChecker.getPermissions(FsPermissionChecker.java:288)
	at org.apache.impala.catalog.HdfsTable.getAvailableAccessLevel(HdfsTable.java:787)
	at org.apache.impala.catalog.HdfsTable.createPartition(HdfsTable.java:861)
	at org.apache.impala.catalog.HdfsTable.loadAllPartitions(HdfsTable.java:639)
	at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1065)
	at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1002)
	at org.apache.impala.catalog.TableLoader.load(TableLoader.java:80)
	at org.apache.impala.catalog.TableLoadingMgr$2.call(TableLoadingMgr.java:237)
	at org.apache.impala.catalog.TableLoadingMgr$2.call(TableLoadingMgr.java:234)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
W0531 23:09:45.538826 92906 RetryInvocationHandler.java:126] Exception while invoking class org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getAclStatus over null. Not retrying because try once and fail.
Java exception follows:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.AclException): The ACL operation has been rejected.  Support for ACLs has been disabled by setting dfs.namenode.acls.enabled to false.
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAclsConfigFlag(FSNamesystem.java:9637)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAclStatus(FSNamesystem.java:9111)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getAclStatus(NameNodeRpcServer.java:1620)
	at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getAclStatus(AuthorizationProviderProxyClientProtocol.java:907)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getAclStatus(ClientNamenodeProtocolServerSideTranslatorPB.java:1335)
	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2217)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2213)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2211)

	at org.apache.hadoop.ipc.Client.call(Client.java:1502)
	at org.apache.hadoop.ipc.Client.call(Client.java:1439)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
	at com.sun.proxy.$Proxy17.getAclStatus(Unknown Source)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getAclStatus(ClientNamenodeProtocolTranslatorPB.java:1336)
	at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:258)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
	at com.sun.proxy.$Proxy18.getAclStatus(Unknown Source)
	at org.apache.hadoop.hdfs.DFSClient.getAclStatus(DFSClient.java:3353)
	at org.apache.hadoop.hdfs.DistributedFileSystem$42.doCall(DistributedFileSystem.java:2059)
	at org.apache.hadoop.hdfs.DistributedFileSystem$42.doCall(DistributedFileSystem.java:2056)
	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
	at org.apache.hadoop.hdfs.DistributedFileSystem.getAclStatus(DistributedFileSystem.java:2056)
	at org.apache.impala.util.FsPermissionChecker.getPermissions(FsPermissionChecker.java:288)
	at org.apache.impala.catalog.HdfsTable.getAvailableAccessLevel(HdfsTable.java:787)
	at org.apache.impala.catalog.HdfsTable.createPartition(HdfsTable.java:861)
	at org.apache.impala.catalog.HdfsTable.loadAllPartitions(HdfsTable.java:639)
	at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1065)
	at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1002)
	at org.apache.impala.catalog.TableLoader.load(TableLoader.java:80)
	at org.apache.impala.catalog.TableLoadingMgr$2.call(TableLoadingMgr.java:237)
	at org.apache.impala.catalog.TableLoadingMgr$2.call(TableLoadingMgr.java:234)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. IMPALA-4162 Extensive logging in HDFS NameNode during metadata load when dfs.namenode.acls.enabled=false

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              bharathv Bharath Vissapragada
              mmokhtar Mostafa Mokhtar
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: