Uploaded image for project: 'Atlas'
  1. Atlas
  2. ATLAS-2157

Atlas Hive hook runs into unhandled exceptions from beeline if there are temp tables involved

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 0.8-incubating, 0.7.1-incubating
    • 0.8.2, 1.0.0
    • atlas-intg
    • None
    • HDP 2.6.2

    Description

      Opening JIRA as requested by madhan.neethiraj in the mailing list
      Repro steps:
      Insert into a Hive table from beeline, insert goes through fine but Atlas Hive hook throws errors.

      Detailed description pasted from mailing list:
      created a table from both Hive CLI and beeline and inserted a row with
      Atlas Hive hook enabled. From Hive CLI the insert goes through fine without any errors but
      from beeline I see errors for a temp table not found although insert itself succeeds and temp
      table shows up in table listing. Do I need to do anything extra for beeline/HS2 connections
      when Atlas is enabled ?

      hive> create table test1(c1 int, c2 string);
      OK
      Time taken: 2.108 seconds
      hive> insert into test1 values(1,'v');
      Query ID = hive_20170920092542_d8ec0a4b-b88d-4d87-b59d-790f2eae6792
      Total jobs = 1
      Launching Job 1 out of 1
      Status: Running (Executing on YARN cluster with App id application_1505852425742_0004)
       
      --------------------------------------------------------------------------------
              VERTICES      STATUS  TOTAL  COMPLETED  RUNNING  PENDING  FAILED  KILLED
      --------------------------------------------------------------------------------
      Map 1 ..........   SUCCEEDED      1          1        0        0       0       0
      --------------------------------------------------------------------------------
      VERTICES: 01/01  [==========================>>] 100%  ELAPSED TIME: 4.68 s
      --------------------------------------------------------------------------------
      Loading data to table default.test1
      Table default.test1 stats: [numFiles=1, numRows=1, totalSize=4, rawDataSize=3]
      OK
      Time taken: 6.7 seconds
      hive> select * from test1;
      OK
      1       v
      Time taken: 0.112 seconds, Fetched: 1 row(s)
      hive>
      
      Beeline version 1.2.1000.2.6.2.0-133 by Apache Hive
      0: jdbc:hive2://> create table s1(c1 int,c2 string);
      OK
      No rows affected (2.345 seconds)
       
      0: jdbc:hive2://> insert into s1 values(1,'v');
      17/09/20 09:28:24 [main]: ERROR hdfs.KeyProviderCache: Could not find uri with key [dfs.encryption.key.provider.uri]
      to create a keyProvider !!
      Query ID = hive_20170920092824_14bfdec4-dfe8-4b97-b5be-0af74de055ce
      Total jobs = 1
      Launching Job 1 out of 1
      Status: Running (Executing on YARN cluster with App id application_1505852425742_0005)
       
      Loading data to table default.s1
      Table default.s1 stats: [numFiles=1, numRows=1, totalSize=4, rawDataSize=3]
      OK
      No rows affected (498.295 seconds)
      0: jdbc:hive2://>
      0: jdbc:hive2://> 17/09/20 09:36:42 [Atlas Logger 1]: ERROR metadata.Hive: Table values__tmp__table__1 not found: default.values__tmp__table__1 table not found
      17/09/20 09:36:43 [Atlas Logger 1]: ERROR hook.HiveHook: Atlas hook failed due to error
      java.lang.reflect.UndeclaredThrowableException
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1884)
              at org.apache.atlas.hive.hook.HiveHook$2.run(HiveHook.java:195)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      Caused by: org.apache.atlas.hook.AtlasHookException: HiveHook.registerProcess() failed.
              at org.apache.atlas.hive.hook.HiveHook.registerProcess(HiveHook.java:701)
              at org.apache.atlas.hive.hook.HiveHook.collect(HiveHook.java:268)
              at org.apache.atlas.hive.hook.HiveHook.access$200(HiveHook.java:83)
              at org.apache.atlas.hive.hook.HiveHook$2$1.run(HiveHook.java:198)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:422)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
              ... 6 more
      Caused by: org.apache.atlas.hook.AtlasHookException: HiveHook.processHiveEntity() failed.
              at org.apache.atlas.hive.hook.HiveHook.processHiveEntity(HiveHook.java:731)
              at org.apache.atlas.hive.hook.HiveHook.registerProcess(HiveHook.java:668)
              ... 12 more
      Caused by: org.apache.atlas.hook.AtlasHookException: HiveHook.createOrUpdateEntities() failed.
              at org.apache.atlas.hive.hook.HiveHook.createOrUpdateEntities(HiveHook.java:597)
              at org.apache.atlas.hive.hook.HiveHook.processHiveEntity(HiveHook.java:711)
              ... 13 more
      Caused by: org.apache.atlas.hook.AtlasHookException: HiveHook.createOrUpdateEntities() failed.
              at org.apache.atlas.hive.hook.HiveHook.createOrUpdateEntities(HiveHook.java:589)
              at org.apache.atlas.hive.hook.HiveHook.createOrUpdateEntities(HiveHook.java:595)
              ... 14 more
      Caused by: org.apache.hadoop.hive.ql.metadata.InvalidTableException: Table not found values__tmp__table__1
              at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1216)
              at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1186)
              at org.apache.atlas.hive.hook.HiveHook.createOrUpdateEntities(HiveHook.java:568)
              ... 15 more
      0: jdbc:hive2://> select * from s1;
      OK
      +--------+--------+--+
      | s1.c1  | s1.c2  |
      +--------+--------+--+
      | 1      | v      |
      +--------+--------+--+
      1 row selected (0.259 seconds)
      0: jdbc:hive2://>
      0: jdbc:hive2://> show tables;
      OK
      +------------------------+--+
      |        tab_name        |
      +------------------------+--+
      | s1                     |
      | test1                  |
      | values__tmp__table__1  |
      +------------------------+--+
      3 rows selected (0.075 seconds)
      

      Attachments

        1. ATLAS-2157.patch
          9 kB
          Madhan Neethiraj

        Activity

          People

            madhan Madhan Neethiraj
            sailajanch Sailaja Navvluru
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: