[IMPALA-9664] Support Hive replication for ACID tables

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: Impala 4.0.0
• Epic Color: ghx-label-10
• Assignee: Vihang Karajgaonkar
• Reporter: Vihang Karajgaonkar
• Created: 16/Apr/20 21:43
• Updated: 07/Oct/20 14:07
• Resolved: 29/Sep/20 16:02

Description

According to what we see in Hive source code, for transactional tables, the insert events are fired with a different API addWriteNotificationLog. Currently Impala fires firelistenerEvent for both transactional and non-transactional tables. We should look at what is the difference between the two APIs and see if we need to handle transactional tables differently.

References:
https://github.com/apache/hive/blob/c3afb57bdb1041f566fbbd896f625328fc9656a0/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L2402

https://github.com/apache/hive/blob/c3afb57bdb1041f566fbbd896f625328fc9656a0/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L2236

These insert events are used to replicate the changes in the ACID tables by tools like Hive replication. With the ability of insert data into ACID tables from Impala, we should also generate the insert events appropriately so that the replication works seemlessly. Additionally, the truncate table command should use the HMS API to truncate the table instead of deleteing the files directly from filesystem since it takes care of moving the files to a replication change management directory so that replication can have access to dropped data files.

Note that for external tables, Hive replication doesn't need to keep track of the files. It only replicates the table metadata based on events and the data files are "distcped" to the target cluster.

Issue Links

• is duplicated by: IMPALA-10135 Insert events doesn't contain the inserted data files