Uploaded image for project: 'Atlas'
  1. Atlas
  2. ATLAS-3608

Hive Bridge: Hive Metastore: Alter Table Query Not Handled Correctly

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.0, trunk
    • Component/s: atlas-core
    • Labels:
      None

      Description

      Background

      DDL queries in Impala are processed via Hive Metastore bridge.

       

      Steps to Duplicate

      Keep HMS logs in view. Depending on installation, they can be found at this location: /var/log/hive/hadoop-cmf-HIVE-1HIVEMETASTORE<host name>.log.out

      From Impala:

      • Run impala-shell
      • Run create database stocks; use stocks; create table daily (dt string, open string, high string); create view daily_rpt as select * from daily; create external table weekly (dt string, open string, high string);
      • Note within Atlas that the new entities for stocks, daily, daily_rpt and weekly have been created. Note the columns in weekly table.
      • From impala-shell, run alter table weekly add columns ( newCol string);

      Expected results

      • HMS logs should not show NullPointerException.
      • Atlas should show the table weekly with the newCol column.

       

      Observed results:

      HMS logs show NullPointerExcetion from Atlas hook.
      New entity newCol is not seen within Atlas.

      Root cause

      When assessing the incoming event to determine the type of alter, Atlas uses table parameter. The recent build has a new parameter for timestamp: last_modified_time transient_lastDdlTime. This results in incorrect assessment. Hence the alter event is incorrectly processed, thereby causing an exception.

        Attachments

          Activity

            People

            • Assignee:
              amestry Ashutosh Mestry
              Reporter:
              amestry Ashutosh Mestry
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Due:
                Created:
                Updated:
                Resolved: