Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Fixed
-
2.1.0
-
None
Description
Data move codepath is broken with hive 2.1.0-SNAPSHOT with hadoop 2.8.0-snapshot.
Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): Path not found: /apps/hive/warehouse/tpcds_bin_partitioned_orc_10000.db/date_dim1 at org.apache.hadoop.hdfs.server.namenode.FSDirEncryptionZoneOp.getEZForPath(FSDirEncryptionZoneOp.java:178) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getEZForPath(FSNamesystem.java:7336) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getEZForPath(NameNodeRpcServer.java:1973) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getEZForPath(ClientNamenodeProtocolServerSideTranslatorPB.java:1376) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:645) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2339) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2335) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1711) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2333) at org.apache.hadoop.ipc.Client.call(Client.java:1448) at org.apache.hadoop.ipc.Client.call(Client.java:1385) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230) at com.sun.proxy.$Proxy30.getEZForPath(Unknown Source)/apps/hive/warehouse/tpcds_bin_partitioned_orc_200.db/ ... ... ... 2016-05-11T09:40:43,760 ERROR [main]: ql.Driver (:()) - FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask. Unable to move source hdfs://xyz:8020/apps/hive/warehouse/tpcds_bin_partitioned_orc_10000.db/.hive-staging_hive_2016-05-11_09-40-42_489_5056654133706433454-1/-ext-10002 to destination hdfs://xyz:8020/apps/hive/warehouse/tpcds_bin_partitioned_orc_10000.db/date_dim1
hdfsEncryptionShim.isPathEncrypted(destf) in Hive could end up throwing FileNotFoundException as the destf is not present yet. This causes moveFile to fail.