Uploaded image for project: 'Atlas'
  1. Atlas
  2. ATLAS-3033

ATLAS Hive hook fails for CTAS query involving UDFs

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 0.8.3
    • 0.8.4, 1.2.0, 2.0.0
    • None
    • None

    Description

      ATLAS Hive Hook fails to notify for CTAS queries involving UDFs, Hook fails with the exception as below

      org.apache.hadoop.hive.ql.metadata.InvalidTableException: Table not found _dummy_table
      

      complete stack trace.

      2018-12-26 12:15:22,234 ERROR [HiveServer2-Background-Pool: Thread-339]: metadata.Hive (Hive.java:getTable(1220)) - Table _dummy_table not found: _dummy_database._
      dummy_table table not found
      2018-12-26 12:15:22,234 ERROR [HiveServer2-Background-Pool: Thread-339]: hook.HiveHook (HiveHook.java:run(160)) - HiveHook.run(): failed to process operation CREAT
      ETABLE_AS_SELECT
      org.apache.hadoop.hive.ql.metadata.InvalidTableException: Table not found _dummy_table
              at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1221)
              at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1191)
              at org.apache.atlas.hive.hook.events.BaseHiveEvent.toAtlasEntity(BaseHiveEvent.java:225)
              at org.apache.atlas.hive.hook.events.BaseHiveEvent.getInputOutputEntity(BaseHiveEvent.java:203)
              at org.apache.atlas.hive.hook.events.CreateHiveProcess.getEntities(CreateHiveProcess.java:84)
              at org.apache.atlas.hive.hook.events.CreateHiveProcess.getNotificationMessages(CreateHiveProcess.java:59)
              at org.apache.atlas.hive.hook.HiveHook.run(HiveHook.java:157)
              at org.apache.atlas.hive.hook.HiveHook.run(HiveHook.java:52)
              at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1619)
              at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1303)
              at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1170)
              at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1165)
              at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:197)
              at org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:76)
              at org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:255)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:422)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
                      at org.apache.hive.service.cli.operation.SQLOperation$2.run(SQLOperation.java:266)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      2018-12-26 12:15:22,234 INFO  [HiveServer2-Background-Pool: Thread-339]: log.PerfLogger (PerfLogger.java:PerfLogEnd(177)) - </PERFLOG method=PostHook.org.apache.atlas.hive.hook.HiveHook start=1545826522229 end=1545826522234 duration=5 from=org.apache.hadoop.hive.ql.Driver>
      

      The problem looks to be appearing as result being stored for UDF in temporary table _dummy_database._dummy_table and while creating a Lineage the process fails.

      Steps to reproduce is running below SQL.

      create table masktest11 as select stack(2, "eddie", "111", "ravi", "222") as (name, ssn);
      

      Attachments

        1. ATLAS-3033.patch
          2 kB
          Nixon Rodrigues
        2. ATLAS-3033-branch-2.0.patch
          7 kB
          Madhan Neethiraj
        3. ATLAS-3033-master.patch
          7 kB
          Madhan Neethiraj

        Activity

          People

            nixon Nixon Rodrigues
            nixon Nixon Rodrigues
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: