[HIVE-18530] Replication should skip MM table (for now)

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0
• Component/s: repl
• Target Version/s: 3.0.0
• Hadoop Flags: Reviewed
• Assignee: Daniel Dai
• Reporter: Daniel Dai
• Created: 24/Jan/18 23:20
• Updated: 22/May/18 23:14
• Resolved: 26/Jan/18 23:45

Description

Currently replication cannot handle transactional table (including MM table) until HIVE-18320. HIVE-17504 skips table with transactional=true explicitly. HIVE-18352 changes the logic to use AcidUtils.isAcidTable for the same purpose. However, isAcidTable returns false for mm table, thus Hive still dump mm table during replication.
Here is an error message during dump mm table:

ERROR : FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.repl.ReplDumpTask. java.io.FileNotFoundException: Path is not a file: /apps/hive/warehouse/testrepldb5.db/test1/delta_0000261_0000261_0000
	at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:89)
	at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:75)
	at org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getBlockLocations(FSDirStatAndListingOp.java:154)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1920)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:731)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:424)
	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1965)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)

INFO  : Completed executing command(queryId=hive_20180119203438_293813df-7630-47fa-bc30-5ef7cbb42842); Time taken: 1.119 seconds
Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.repl.ReplDumpTask. java.io.FileNotFoundException: Path is not a file: /apps/hive/warehouse/testrepldb5.db/test1/delta_0000261_0000261_0000
	at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:89)
	at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:75)
	at org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getBlockLocations(FSDirStatAndListingOp.java:154)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1920)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:731)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:424)
	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1965)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675) (state=08S01,code=1)
0: jdbc:hive2://ctr-e137-1514896590304-25219-> Closing: 0: jdbc:hive2://ctr-e137-1514896590304-25219-02-000005.hwx.site:2181,ctr-e137-1514896590304-25219-02-000012.hwx.site:2181,ctr-e137-1514896590304-25219-02-000009.hwx.site:2181,ctr-e137-1514896590304-25219-02-000004.hwx.site:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;principal=hive/[e-mail]

We shall switch to use AcidUtils.isTransactionalTable.