Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-10098

HS2 local task for map join fails in KMS encrypted cluster

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 1.2.0
    • Security
    • None

    Description

      Env: KMS was enabled after cluster was kerberos secured.
      Problem: PROBLEM: Any Hive query via beeline that performs a MapJoin fails with a java.lang.reflect.UndeclaredThrowableException from KMSClientProvider.addDelegationTokens.

      2015-03-18 08:49:17,948 INFO [main]: Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir 
      2015-03-18 08:49:19,048 WARN [main]: security.UserGroupInformation (UserGroupInformation.java:doAs(1645)) - PriviledgedActionException as:hive (auth:KERBEROS) cause:org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) 
      2015-03-18 08:49:19,050 ERROR [main]: mr.MapredLocalTask (MapredLocalTask.java:executeFromChildJVM(314)) - Hive Runtime Error: Map local work failed 
      java.io.IOException: java.io.IOException: java.lang.reflect.UndeclaredThrowableException 
      at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:634) 
      at org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.startForward(MapredLocalTask.java:363) 
      at org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.startForward(MapredLocalTask.java:337) 
      at org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.executeFromChildJVM(MapredLocalTask.java:303)
      at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:735) 
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
      at java.lang.reflect.Method.invoke(Method.java:606) 
      at org.apache.hadoop.util.RunJar.main(RunJar.java:212) 
      Caused by: java.io.IOException: java.lang.reflect.UndeclaredThrowableException 
      at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:826) 
      at org.apache.hadoop.crypto.key.KeyProviderDelegationTokenExtension.addDelegationTokens(KeyProviderDelegationTokenExtension.java:86) 
      at org.apache.hadoop.hdfs.DistributedFileSystem.addDelegationTokens(DistributedFileSystem.java:2017) 
      at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:121) 
      at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:100) 
      at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:80) 
      at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:205) 
      at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:313) 
      at org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:413) 
      at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:559) 
      ... 9 more 
      Caused by: java.lang.reflect.UndeclaredThrowableException 
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1655) 
      at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:808) 
      ... 18 more 
      Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) 
      at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:306) 
      at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:196) 
      at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:127)
      

      To make sure map join happen, test need a small table join with a large one, for example:

      CREATE TABLE if not exists jsmall (code string, des string, t int, s int) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t';
      CREATE TABLE if not exists jbig1 (code string, des string, t int, s int) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t';
      load data local inpath '/tmp/jdata' into table jsmall;
      load data local inpath '/tmp/jdata' into table jbig1;
      load data local inpath '/tmp/jdata' into table jbig1;
      load data local inpath '/tmp/jdata' into table jbig1;
      load data local inpath '/tmp/jdata' into table jbig1;
      load data local inpath '/tmp/jdata' into table jbig1;
      load data local inpath '/tmp/jdata' into table jbig1;
      load data local inpath '/tmp/jdata' into table jbig1;
      
      select count(*) from jsmall small join jbig1 big on (small.code = big.code);
      

      Attachments

        1. HIVE-10098.1.patch
          1 kB
          Yongzhi Chen
        2. HIVE-10098.2.patch
          1 kB
          Yongzhi Chen

        Activity

          People

            ychena Yongzhi Chen
            ychena Yongzhi Chen
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: