Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-9055

HDFS Caching with Impala: Expiration 26687997791:19:48:13.951 exceeds the max relative expiration time of <maxTtl>

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • Impala 3.4.0
    • Catalog
    • None
    • ghx-label-7

    Description

      HDFS Caching with Impala:
      If we create a pool specifying the maxTtl with the hdfs command:
      e.g:
      sudo -u hdfs hdfs cacheadmin -addPool foo -owner impala -group hdfs -mode 755 -limit 100000000000 -maxTtl 7d

      when we try to alter a table adding a partition in Impala:
      e.g:

      alter table foo partition (p1=1) set cached in 'foo'
      

      we get a failure with the exception:

      ERROR: ImpalaRuntimeException: Expiration 26687997791:19:48:13.951 exceeds the max relative expiration time of 604800000 ms.
      	at org.apache.hadoop.hdfs.server.namenode.CacheManager.validateExpiryTime(CacheManager.java:378)
      	at org.apache.hadoop.hdfs.server.namenode.CacheManager.addDirective(CacheManager.java:528)
      	at org.apache.hadoop.hdfs.server.namenode.FSNDNCacheOp.addCacheDirective(FSNDNCacheOp.java:45)
      	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.addCacheDirective(FSNamesystem.java:6782)
      	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addCacheDirective(NameNodeRpcServer.java:1883)
      	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addCacheDirective(ClientNamenodeProtocolServerSideTranslatorPB.java:1265)
      	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
      	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
      	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
      	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
      	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:422)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1685)
      	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
      
      CAUSED BY: InvalidRequestException: Expiration 26687997791:19:48:13.951 exceeds the max relative expiration time of 604800000 ms.
      	at org.apache.hadoop.hdfs.server.namenode.CacheManager.validateExpiryTime(CacheManager.java:378)
      	at org.apache.hadoop.hdfs.server.namenode.CacheManager.addDirective(CacheManager.java:528)
      	at org.apache.hadoop.hdfs.server.namenode.FSNDNCacheOp.addCacheDirective(FSNDNCacheOp.java:45)
      	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.addCacheDirective(FSNamesystem.java:6782)
      	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addCacheDirective(NameNodeRpcServer.java:1883)
      	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addCacheDirective(ClientNamenodeProtocolServerSideTranslatorPB.java:1265)
      	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
      	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
      	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
      	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
      	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:422)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1685)
      	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
      
      CAUSED BY: RemoteException: Expiration 26687997791:19:48:13.951 exceeds the max relative expiration time of 604800000 ms.
      	at org.apache.hadoop.hdfs.server.namenode.CacheManager.validateExpiryTime(CacheManager.java:378)
      	at org.apache.hadoop.hdfs.server.namenode.CacheManager.addDirective(CacheManager.java:528)
      	at org.apache.hadoop.hdfs.server.namenode.FSNDNCacheOp.addCacheDirective(FSNDNCacheOp.java:45)
      	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.addCacheDirective(FSNamesystem.java:6782)
      	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addCacheDirective(NameNodeRpcServer.java:1883)
      	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addCacheDirective(ClientNamenodeProtocolServerSideTranslatorPB.java:1265)
      	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
      	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
      	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
      	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
      	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:422)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1685)
      	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
      

      this is always reproducible.

      The workaround is to do not set the maxTtl in the hdfs command during the pool creation.

      Here's the repro steps:

      impala-shell -i <ImpalaD hostname> -q "drop table if exists foo;"
      impala-shell -i <ImpalaD hostname> -q "create table if not exists foo (c1 int, c2 string) partitioned by (p1 int) stored as parquet location '/user/hive/warehouse/foo';"
      impala-shell -i <ImpalaD hostname> -q "insert into foo partition (p1 = 1 ) values  (1 ,'one');"
      sudo -u hdfs hdfs cacheadmin -removePool foo
      sudo -u hdfs hdfs cacheadmin -listDirectives -stats
      sudo -u hdfs hdfs cacheadmin -listPools -stats
      sudo -u hdfs hdfs cacheadmin -addPool foo -owner impala -group hdfs -mode 755 -limit 100000000000  -maxTtl 7d
      sudo -u hdfs hdfs cacheadmin -addDirective -path /user/hive/warehouse/foo -pool foo -ttl 7d
      sudo -u hdfs hdfs cacheadmin -listDirectives -stats
      sudo -u hdfs hdfs cacheadmin -listPools -stats
      impala-shell -i <ImpalaD hostname> -q "alter table foo set uncached;" 
      sleep 5;
      impala-shell -i <ImpalaD hostname> -q "alter table foo partition (p1=1) set cached in 'foo';" 
      

      I did not found any already open jira for this issue that looks reproducible in many CDH versions, I appreciate if you can take a look on it.

      Attachments

        Activity

          People

            boroknagyz Zoltán Borók-Nagy
            adrenas Adriano
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: