[ATLAS-3033] ATLAS Hive hook fails for CTAS query involving UDFs

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.8.3
• Fix Version/s: 0.8.4, 1.2.0, 2.0.0
• Assignee: Nixon Rodrigues
• Reporter: Nixon Rodrigues
• Created: 18/Jan/19 12:40
• Updated: 03/Aug/21 05:59
• Resolved: 29/Jan/19 10:32

Description

ATLAS Hive Hook fails to notify for CTAS queries involving UDFs, Hook fails with the exception as below

org.apache.hadoop.hive.ql.metadata.InvalidTableException: Table not found _dummy_table

complete stack trace.

2018-12-26 12:15:22,234 ERROR [HiveServer2-Background-Pool: Thread-339]: metadata.Hive (Hive.java:getTable(1220)) - Table _dummy_table not found: _dummy_database._
dummy_table table not found
2018-12-26 12:15:22,234 ERROR [HiveServer2-Background-Pool: Thread-339]: hook.HiveHook (HiveHook.java:run(160)) - HiveHook.run(): failed to process operation CREAT
ETABLE_AS_SELECT
org.apache.hadoop.hive.ql.metadata.InvalidTableException: Table not found _dummy_table
        at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1221)
        at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1191)
        at org.apache.atlas.hive.hook.events.BaseHiveEvent.toAtlasEntity(BaseHiveEvent.java:225)
        at org.apache.atlas.hive.hook.events.BaseHiveEvent.getInputOutputEntity(BaseHiveEvent.java:203)
        at org.apache.atlas.hive.hook.events.CreateHiveProcess.getEntities(CreateHiveProcess.java:84)
        at org.apache.atlas.hive.hook.events.CreateHiveProcess.getNotificationMessages(CreateHiveProcess.java:59)
        at org.apache.atlas.hive.hook.HiveHook.run(HiveHook.java:157)
        at org.apache.atlas.hive.hook.HiveHook.run(HiveHook.java:52)
        at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1619)
        at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1303)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1170)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1165)
        at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:197)
        at org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:76)
        at org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:255)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
                at org.apache.hive.service.cli.operation.SQLOperation$2.run(SQLOperation.java:266)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
2018-12-26 12:15:22,234 INFO  [HiveServer2-Background-Pool: Thread-339]: log.PerfLogger (PerfLogger.java:PerfLogEnd(177)) - </PERFLOG method=PostHook.org.apache.atlas.hive.hook.HiveHook start=1545826522229 end=1545826522234 duration=5 from=org.apache.hadoop.hive.ql.Driver>

The problem looks to be appearing as result being stored for UDF in temporary table _dummy_database._dummy_table and while creating a Lineage the process fails.

Steps to reproduce is running below SQL.

create table masktest11 as select stack(2, "eddie", "111", "ravi", "222") as (name, ssn);