Description
ATLAS Hive Hook fails to notify for CTAS queries involving UDFs, Hook fails with the exception as below
org.apache.hadoop.hive.ql.metadata.InvalidTableException: Table not found _dummy_table
complete stack trace.
2018-12-26 12:15:22,234 ERROR [HiveServer2-Background-Pool: Thread-339]: metadata.Hive (Hive.java:getTable(1220)) - Table _dummy_table not found: _dummy_database._ dummy_table table not found 2018-12-26 12:15:22,234 ERROR [HiveServer2-Background-Pool: Thread-339]: hook.HiveHook (HiveHook.java:run(160)) - HiveHook.run(): failed to process operation CREAT ETABLE_AS_SELECT org.apache.hadoop.hive.ql.metadata.InvalidTableException: Table not found _dummy_table at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1221) at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1191) at org.apache.atlas.hive.hook.events.BaseHiveEvent.toAtlasEntity(BaseHiveEvent.java:225) at org.apache.atlas.hive.hook.events.BaseHiveEvent.getInputOutputEntity(BaseHiveEvent.java:203) at org.apache.atlas.hive.hook.events.CreateHiveProcess.getEntities(CreateHiveProcess.java:84) at org.apache.atlas.hive.hook.events.CreateHiveProcess.getNotificationMessages(CreateHiveProcess.java:59) at org.apache.atlas.hive.hook.HiveHook.run(HiveHook.java:157) at org.apache.atlas.hive.hook.HiveHook.run(HiveHook.java:52) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1619) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1303) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1170) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1165) at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:197) at org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:76) at org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869) at org.apache.hive.service.cli.operation.SQLOperation$2.run(SQLOperation.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2018-12-26 12:15:22,234 INFO [HiveServer2-Background-Pool: Thread-339]: log.PerfLogger (PerfLogger.java:PerfLogEnd(177)) - </PERFLOG method=PostHook.org.apache.atlas.hive.hook.HiveHook start=1545826522229 end=1545826522234 duration=5 from=org.apache.hadoop.hive.ql.Driver>
The problem looks to be appearing as result being stored for UDF in temporary table _dummy_database._dummy_table and while creating a Lineage the process fails.
Steps to reproduce is running below SQL.
create table masktest11 as select stack(2, "eddie", "111", "ravi", "222") as (name, ssn);