Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-15128

"hive.exec.stagingdir" produces an "AccessControlException" when working with temp tables

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.1.0
    • Fix Version/s: None
    • Component/s: Beeline, CLI, Hive
    • Labels:
      None
    • Environment:

      Hive 2.1.0, Hadoop 2.7.2

      Description

      The property "hive.exec.stagingdir" is producing an AccessControlException error, only when working with a temporary table and with an INSERT INTO TABLE statement.
      The configuration used in the hive-site.xml is:

        <property>
            <name>hive.exec.stagingdir</name>
            <value>/user/${user.name}/.hiveStaging/.staging</value>
            <description>In Hive >= 0.14, set to ${hive.exec.scratchdir}/${user.name}/.staging</description>
        </property>
      

      Trying to insert a value into a temporary table will produce the following output:

      hive (shfs3453)> INSERT INTO TABLE test_table_tmp VALUES (11101, 'John', 'Oakland');
      Query ID = shfs3453_20161103180522_5fceae9b-6fb5-4cdd-a0d9-37cc93b27a9f
      Total jobs = 1
      Launching Job 1 out of 1
      
      
      Status: Running (Executing on YARN cluster with App id application_1478096256452_0018)
      
      ----------------------------------------------------------------------------------------------
              VERTICES      MODE        STATUS  TOTAL  COMPLETED  RUNNING  PENDING  FAILED  KILLED
      ----------------------------------------------------------------------------------------------
      Map 1 .......... container     SUCCEEDED      1          1        0        0       0       0
      ----------------------------------------------------------------------------------------------
      VERTICES: 01/01  [==========================>>] 100%  ELAPSED TIME: 16.09 s
      ----------------------------------------------------------------------------------------------
      Loading data to table shfs3453.test_table_tmp
      Failed with exception org.apache.hadoop.security.AccessControlException: User does not belong to hadoop
              at org.apache.hadoop.hdfs.server.namenode.FSDirAttrOp.setOwner(FSDirAttrOp.java:86)
              at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setOwner(FSNamesystem.java:1676)
              at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.setOwner(NameNodeRpcServer.java:702)
              at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.setOwner(ClientNamenodeProtocolServerSideTranslatorPB.java:464)
              at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
              at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
              at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
              at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
              at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:415)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
              at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)
      
      FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask. org.apache.hadoop.security.AccessControlException: User does not belong to hadoop
              at org.apache.hadoop.hdfs.server.namenode.FSDirAttrOp.setOwner(FSDirAttrOp.java:86)
              at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setOwner(FSNamesystem.java:1676)
              at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.setOwner(NameNodeRpcServer.java:702)
              at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.setOwner(ClientNamenodeProtocolServerSideTranslatorPB.java:464)
              at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
              at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
              at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
              at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
              at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:415)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
              at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)
      

      But if you do an INSERT OVERWRITE TABLE, there will be a warning only. No AccessControlException error here and erverything is properly inserted.

      hive (shfs3453)> INSERT OVERWRITE TABLE test_table_tmp VALUES (11101, 'John', 'Oakland');
      Query ID = shfs3453_20161103180603_d8d5f689-cc55-44d0-b160-bba0df8d186a
      Total jobs = 1
      Launching Job 1 out of 1
      
      
      Status: Running (Executing on YARN cluster with App id application_1478096256452_0018)
      
      ----------------------------------------------------------------------------------------------
              VERTICES      MODE        STATUS  TOTAL  COMPLETED  RUNNING  PENDING  FAILED  KILLED
      ----------------------------------------------------------------------------------------------
      Map 1 .......... container     SUCCEEDED      2          2        0        0       0       0
      ----------------------------------------------------------------------------------------------
      VERTICES: 01/01  [==========================>>] 100%  ELAPSED TIME: 6.74 s
      ----------------------------------------------------------------------------------------------
      Loading data to table shfs3453.test_table_tmp
      chgrp: changing ownership of 'hdfs://sandbox/tmp/hive/shfs3453/edab56b7-b1e1-4f87-bdfd-eeece41f6fc9/_tmp_space.db/874bfdbd-7bab-4e0c-85ac-ec4552ea560b': User does not belong to hadoop
      OK
      Time taken: 8.224 seconds
      

      If the previous query is done one more time, the warning will not be displayed.

      hive (shfs3453)> INSERT OVERWRITE TABLE test_table_tmp VALUES (11101, 'John', 'Oakland');
      Query ID = shfs3453_20161103180831_40350dc3-1f66-45e2-8304-90b00933aead
      Total jobs = 1
      Launching Job 1 out of 1
      
      
      Status: Running (Executing on YARN cluster with App id application_1478096256452_0018)
      
      ----------------------------------------------------------------------------------------------
              VERTICES      MODE        STATUS  TOTAL  COMPLETED  RUNNING  PENDING  FAILED  KILLED
      ----------------------------------------------------------------------------------------------
      Map 1 .......... container     SUCCEEDED      1          1        0        0       0       0
      ----------------------------------------------------------------------------------------------
      VERTICES: 01/01  [==========================>>] 100%  ELAPSED TIME: 13.00 s
      ----------------------------------------------------------------------------------------------
      Loading data to table shfs3453.test_table_tmp
      OK
      Time taken: 14.161 seconds
      

      Note: if the warning is displayed one time, an INSERT INTO TABLE will work.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              BigDataOrange Alexandre Linte
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: