DDL queries in Impala are processed via Hive Metastore bridge.
Steps to Duplicate
Keep HMS logs in view. Depending on installation, they can be found at this location: /var/log/hive/hadoop-cmf-HIVE-1
- Run impala-shell
- Run create database stocks; use stocks; create table daily (dt string, open string, high string); create view daily_rpt as select * from daily; create external table weekly (dt string, open string, high string);
- Note within Atlas that the new entities for stocks, daily, daily_rpt and weekly have been created. Note the columns in weekly table.
- From impala-shell, run alter table weekly add columns ( newCol string);
- HMS logs should not show NullPointerException.
- Atlas should show the table weekly with the newCol column.
HMS logs show NullPointerExcetion from Atlas hook.
New entity newCol is not seen within Atlas.
When assessing the incoming event to determine the type of alter, Atlas uses table parameter. The recent build has a new parameter for timestamp: last_modified_time transient_lastDdlTime. This results in incorrect assessment. Hence the alter event is incorrectly processed, thereby causing an exception.