[ATLAS-3608] Hive Bridge: Hive Metastore: Alter Table Query Not Handled Correctly

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.0, trunk
• Component/s: atlas-core
• Assignee: Ashutosh Mestry
• Reporter: Ashutosh Mestry
• Due: 05/Feb/20
• Created: 05/Feb/20 16:28
• Updated: 06/Jul/21 23:10
• Resolved: 06/Feb/20 23:43

Description

Background

DDL queries in Impala are processed via Hive Metastore bridge.

 

Steps to Duplicate

Keep HMS logs in view. Depending on installation, they can be found at this location: /var/log/hive/hadoop-cmf-HIVE-1HIVEMETASTORE<host name>.log.out

From Impala:

• Run impala-shell

• Run create database stocks; use stocks; create table daily (dt string, open string, high string); create view daily_rpt as select * from daily; create external table weekly (dt string, open string, high string);
• Note within Atlas that the new entities for stocks, daily, daily_rpt and weekly have been created. Note the columns in weekly table.
• From impala-shell, run alter table weekly add columns ( newCol string);

Expected results

• HMS logs should not show NullPointerException.
• Atlas should show the table weekly with the newCol column.

 

Observed results:

HMS logs show NullPointerExcetion from Atlas hook.
New entity newCol is not seen within Atlas.

Root cause

When assessing the incoming event to determine the type of alter, Atlas uses table parameter. The recent build has a new parameter for timestamp: last_modified_time transient_lastDdlTime. This results in incorrect assessment. Hence the alter event is incorrectly processed, thereby causing an exception.