Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-14686 Ensure authoritative security coprocessors execute in the correct context
  3. HBASE-14605

Split fails due to 'No valid credentials' error when SecureBulkLoadEndpoint#start tries to access hdfs

    XMLWordPrintableJSON

    Details

    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      When split is requested by non-super user, split related notifications for Coprocessor are executed using the login of the request user.
      Previously the notifications were carried out as super user.
      Show
      When split is requested by non-super user, split related notifications for Coprocessor are executed using the login of the request user. Previously the notifications were carried out as super user.

      Description

      During recent testing in secure cluster (with HBASE-14475), we found the following when user X (non-super user) split a table with region replica:

      2015-10-12 10:58:18,955 ERROR [FifoRpcScheduler.handler1-thread-9] master.HMaster: Region server hbase-4-4.novalocal,60020,1444645588137 reported a fatal error:
      ABORTING region server hbase-4-4.novalocal,60020,1444645588137: The coprocessor org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint threw an unexpected   exception
      Cause:
      java.lang.IllegalStateException: Failed to get FileSystem instance
        at org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.start(SecureBulkLoadEndpoint.java:148)
        at org.apache.hadoop.hbase.coprocessor.CoprocessorHost$Environment.startup(CoprocessorHost.java:415)
        at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadInstance(CoprocessorHost.java:257)
        at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadSystemCoprocessors(CoprocessorHost.java:160)
        at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.<init>(RegionCoprocessorHost.java:192)
        at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:701)
        at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:608)
      ...
      Caused by: java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid          credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "hbase-4-4/172.22.66.186"; destination host is: "os-r6-      okarus-hbase-4-2.novalocal":8020;
        at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
        at org.apache.hadoop.ipc.Client.call(Client.java:1473)
        at org.apache.hadoop.ipc.Client.call(Client.java:1400)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
        at com.sun.proxy.$Proxy18.mkdirs(Unknown Source)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.mkdirs(ClientNamenodeProtocolTranslatorPB.java:555)
        at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
        at com.sun.proxy.$Proxy19.mkdirs(Unknown Source)
        at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:2775)
        at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2746)
        at org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:967)
        at org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:963)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
      

      The cause was that SecureBulkLoadEndpoint#start tried to create staging dir in hdfs as user X but didn't pass authentication.

        Attachments

        1. 14605-branch-1-addendum.txt
          2 kB
          Ted Yu
        2. 14605-0.98-v5.txt
          15 kB
          Ted Yu
        3. 14605-branch-1.0-v5.txt
          15 kB
          Ted Yu
        4. 14605-branch-1-v5.txt
          18 kB
          Ted Yu
        5. 14605-v5.txt
          17 kB
          Ted Yu
        6. 14605-branch-1-v4.txt
          18 kB
          Ted Yu
        7. 14605-v4.txt
          17 kB
          Ted Yu
        8. 144605-branch-1-v3.txt
          17 kB
          Ted Yu
        9. 14605-v3.txt
          16 kB
          Ted Yu
        10. 14605-v3.txt
          16 kB
          Ted Yu
        11. 14605-v3.txt
          16 kB
          Ted Yu
        12. 14605-v2.txt
          16 kB
          Ted Yu
        13. 14605.alt
          16 kB
          Ted Yu
        14. 14605-v1.txt
          2 kB
          Ted Yu

          Issue Links

            Activity

              People

              • Assignee:
                yuzhihong@gmail.com Ted Yu
                Reporter:
                yuzhihong@gmail.com Ted Yu
              • Votes:
                0 Vote for this issue
                Watchers:
                12 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: