Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-21982

HBase Kerberos with no Hadoop/HDFS fails on startup

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Not A Bug
    • Affects Version/s: 1.4.2, 1.4.9
    • Fix Version/s: None
    • Component/s: master, regionserver, rpc
    • Labels:
      None

      Description

      When attempting to Kerberize an HBase Instance that uses the localFS without Hadoop I noticed that instead of the HBase RegionServer Successfully checking in with the HBase Master it fails stating that it was using SIMPLE authentication vs Kerberos. So I think the real question here is does HBase support running without HDFS/Hadoop for the filesystem in Kerberos Mode or is HDFS required?

      Error on RegionServer:

      3-02 13:09:46,314 DEBUG [regionserver/owlms.hdp.senia.org/10.69.68.21:16020] ipc.BlockingRpcConnection: Connecting to owlms.hdp.senia.org/10.69.68.21:16000
      2019-03-02 13:09:46,315 DEBUG [regionserver/owlms.hdp.senia.org/10.69.68.21:16020] security.UserGroupInformation: PrivilegedAction as:hbase (auth:SIMPLE) from:org.apache.hadoop.hbase.ipc.BlockingRpcConnection.setupIOstreams(BlockingRpcConnection.java:452)
      2019-03-02 13:09:46,315 DEBUG [regionserver/owlms.hdp.senia.org/10.69.68.21:16020] security.AbstractHBaseSaslRpcClient: Creating SASL GSSAPI client. Server's Kerberos principal name is hbase/owlms.hdp.senia.org@HDP.SENIA.ORG
      2019-03-02 13:09:46,318 DEBUG [regionserver/owlms.hdp.senia.org/10.69.68.21:16020] security.HBaseSaslRpcClient: Have sent token of size 635 from initSASLContext.
      2019-03-02 13:09:46,318 DEBUG [regionserver/owlms.hdp.senia.org/10.69.68.21:16020] security.UserGroupInformation: PrivilegedActionException as:hbase (auth:SIMPLE) cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hbase.security.AccessDeniedException): Kerberos principal name does NOT have the expected hostname part: hbase
      2019-03-02 13:09:46,319 DEBUG [regionserver/owlms.hdp.senia.org/10.69.68.21:16020] security.UserGroupInformation: PrivilegedAction as:hbase (auth:SIMPLE) from:org.apache.hadoop.hbase.ipc.BlockingRpcConnection.handleSaslConnectionFailure(BlockingRpcConnection.java:374)
      2019-03-02 13:09:46,319 WARN [regionserver/owlms.hdp.senia.org/10.69.68.21:16020] ipc.BlockingRpcConnection: Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hbase.security.AccessDeniedException): Kerberos principal name does NOT have the expected hostname part: hbase
      2019-03-02 13:09:46,319 DEBUG [regionserver/owlms.hdp.senia.org/10.69.68.21:16020] security.UserGroupInformation: PrivilegedActionException as:hbase (auth:SIMPLE) cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hbase.security.AccessDeniedException): Kerberos principal name does NOT have the expected hostname part: hbase
      2019-03-02 13:09:46,319 DEBUG [regionserver/owlms.hdp.senia.org/10.69.68.21:16020] ipc.FailedServers: Added failed server with address owlms.hdp.senia.org/10.69.68.21:16000 to list caused by org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hbase.security.AccessDeniedException): Kerberos principal name does NOT have the expected hostname part: hbase
      2019-03-02 13:09:46,319 WARN [regionserver/owlms.hdp.senia.org/10.69.68.21:16020] regionserver.HRegionServer: error telling master we are up
      com.google.protobuf.ServiceException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hbase.security.AccessDeniedException): Kerberos principal name does NOT have the expected hostname part: hbase
      at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:335)
      at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$200(AbstractRpcClient.java:94)
      at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:571)
      at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerStartup(RegionServerStatusProtos.java:8982)
      at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2431)
      at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:969)
      at java.lang.Thread.run(Thread.java:748)
      Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hbase.security.AccessDeniedException): Kerberos principal name does NOT have the expected hostname part: hbase
      at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:386)
      at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:94)
      at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:409)
      at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:405)
      at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103)
      at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118)
      at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:422)
      at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:327)
      ... 6 more

      Error on HBase Master:
      2019-03-02 14:14:13,593 DEBUG [RpcServer.reader=3,bindAddress=owlms.hdp.senia.org,port=16000] ipc.RpcServer: RpcServer.listener,port=16000: DISCONNECTING client 10.69.68.21:35620 because read count=-1. Number of active connections: 1
      2019-03-02 14:14:14,615 INFO [owlms:16000.activeMasterManager] master.ServerManager: Waiting on RegionServer count=0 to settle; waited=10325215ms, expecting min=1 server(s), max=NO_LIMIT server(s), timeout=4500ms, lastChange=-10325215ms
      2019-03-02 14:14:15,828 DEBUG [master/owlms.hdp.senia.org/10.69.68.21:16000-SendThread(owlms.hdp.senia.org:2181)] zookeeper.ClientCnxn: Got ping response for sessionid: 0x1693caa4d83009b after 0ms
      2019-03-02 14:14:15,928 DEBUG [owlms:16000.activeMasterManager-SendThread(owlms.hdp.senia.org:2181)] zookeeper.ClientCnxn: Got ping response for sessionid: 0x1693caa4d83009c after 0ms
      2019-03-02 14:14:16,119 INFO [owlms:16000.activeMasterManager] master.ServerManager: Waiting on RegionServer count=0 to settle; waited=10326719ms, expecting min=1 server(s), max=NO_LIMIT server(s), timeout=4500ms, lastChange=-10326719ms
      2019-03-02 14:14:16,590 DEBUG [owlms:16000.activeMasterManager-SendThread(owlms.hdp.senia.org:2181)] zookeeper.ClientCnxn: Got ping response for sessionid: 0x1693caa4d83009e after 0ms
      2019-03-02 14:14:16,595 DEBUG [RpcServer.listener,port=16000] ipc.RpcServer: RpcServer.listener,port=16000: connection from 10.69.68.21:52423; # active connections: 1
      2019-03-02 14:14:16,598 DEBUG [RpcServer.reader=4,bindAddress=owlms.hdp.senia.org,port=16000] ipc.RpcServer: Kerberos principal name is hbase
      2019-03-02 14:14:16,598 DEBUG [RpcServer.reader=4,bindAddress=owlms.hdp.senia.org,port=16000] ipc.RpcServer: RpcServer.listener,port=16000: Caught exception while reading:
      org.apache.hadoop.hbase.security.AccessDeniedException: Kerberos principal name does NOT have the expected hostname part: hbase
      at org.apache.hadoop.hbase.ipc.RpcServer$Connection.saslReadAndProcess(RpcServer.java:1468)
      at org.apache.hadoop.hbase.ipc.RpcServer$Connection.process(RpcServer.java:1788)
      at org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess(RpcServer.java:1769)
      at org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(RpcServer.java:955)
      at org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:725)
      at org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:701)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)
      2019-03-02 14:14:16,598 DEBUG [RpcServer.reader=4,bindAddress=owlms.hdp.senia.org,port=16000] ipc.RpcServer: RpcServer.listener,port=16000: DISCONNECTING client 10.69.68.21:52423 because read count=-1. Number of active connections: 1
      2019-03-02 14:14:17,622 INFO [owlms:16000.activeMasterManager] master.ServerManager: Waiting on RegionServer count=0 to settle; waited=10328222ms, expecting min=1 server(s), max=NO_LIMIT server(s), timeout=4500ms, lastChange=-10328222ms
      2019-03-02 14:14:19,125 INFO [owlms:16000.activeMasterManager] master.ServerManager: Waiting on RegionServer count=0 to settle; waited=10329725ms, expecting min=1 server(s), max=NO_LIMIT server(s), timeout=4500ms, lastChange=-10329725ms
      2019-03-02 14:14:19,602 DEBUG [RpcServer.listener,port=16000] ipc.RpcServer: RpcServer.listener,port=16000: connection from 10.69.68.21:58029; # active connections: 1
      2019-03-02 14:14:19,608 DEBUG [RpcServer.reader=5,bindAddress=owlms.hdp.senia.org,port=16000] ipc.RpcServer: Kerberos principal name is hbase
      2019-03-02 14:14:19,608 DEBUG [RpcServer.reader=5,bindAddress=owlms.hdp.senia.org,port=16000] ipc.RpcServer: RpcServer.listener,port=16000: Caught exception while reading:
      org.apache.hadoop.hbase.security.AccessDeniedException: Kerberos principal name does NOT have the expected hostname part: hbase
      at org.apache.hadoop.hbase.ipc.RpcServer$Connection.saslReadAndProcess(RpcServer.java:1468)
      at org.apache.hadoop.hbase.ipc.RpcServer$Connection.process(RpcServer.java:1788)
      at org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess(RpcServer.java:1769)
      at org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(RpcServer.java:955)
      at org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:725)
      at org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:701)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              gss2002 Greg Senia
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: