Uploaded image for project: 'Kudu'
  1. Kudu
  2. KUDU-2142

Client should resolve the canonical master hostname before connecting

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: 1.5.0
    • Fix Version/s: 1.5.0
    • Component/s: client
    • Labels:

      Description

      When connecting to a secure (Kerberized) Kudu cluster, it's important that the master hostname which the client is created with matches the Kerberos hostname. A lot of tools (e.g. kudu-spark-tools) default to using localhost as the master address. Since the client doesn't canonicalize the master address before connecting, these tools will fail to connect. For example, kudu-spark-tools 1.5 fails to connect to a secure cluster with the following error (notice the localhost master address in the trace):

      Exception in thread "main" java.security.PrivilegedActionException: org.apache.kudu.client.NonRecoverableException: Couldn't find a valid master in (localhost:7051). Exceptions received: [org.apache.kudu.client.NonRecoverableException: Server requires Kerberos, but this client is not authenticated (kinit)]
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:360)
      	at org.apache.kudu.spark.kudu.KuduContext.<init>(KuduContext.scala:76)
      	at org.apache.kudu.spark.tools.Generator$.run(IntegrationTestBigLinkedList.scala:155)
      	at org.apache.kudu.spark.tools.Generator$.main(IntegrationTestBigLinkedList.scala:174)
      	at org.apache.kudu.spark.tools.IntegrationTestBigLinkedList$.main(IntegrationTestBigLinkedList.scala:88)
      	at org.apache.kudu.spark.tools.IntegrationTestBigLinkedList.main(IntegrationTestBigLinkedList.scala)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:755)
      	at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
      	at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
      	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)
      	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
      Caused by: org.apache.kudu.client.NonRecoverableException: Couldn't find a valid master in (localhost:7051). Exceptions received: [org.apache.kudu.client.NonRecoverableException: Server requires Kerberos, but this client is not authenticated (kinit)]
      	at org.apache.kudu.client.ConnectToCluster.incrementCountAndCheckExhausted(ConnectToCluster.java:223)
      	at org.apache.kudu.client.ConnectToCluster.access$000(ConnectToCluster.java:48)
      	at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:304)
      	at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:293)
      	at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280)
      	at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
      	at com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1315)
      	at com.stumbleupon.async.Deferred.doCall(Deferred.java:1286)
      	at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
      	at com.stumbleupon.async.Deferred.callback(Deferred.java:1002)
      	at org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:238)
      	at org.apache.kudu.client.KuduRpc.errback(KuduRpc.java:292)
      	at org.apache.kudu.client.RpcProxy.responseReceived(RpcProxy.java:221)
      	at org.apache.kudu.client.RpcProxy.access$000(RpcProxy.java:60)
      	at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:132)
      	at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:128)
      	at org.apache.kudu.client.Connection.cleanup(Connection.java:677)
      	at org.apache.kudu.client.Connection.exceptionCaught(Connection.java:422)
      	at org.apache.kudu.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
      	at org.apache.kudu.client.Connection.handleUpstream(Connection.java:236)
      	at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
      	at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
      	at org.apache.kudu.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:153)
      	at org.apache.kudu.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
      	at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
      	at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
      	at org.apache.kudu.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:153)
      	at org.apache.kudu.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
      	at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
      	at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
      	at org.apache.kudu.shaded.org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:60)
      	at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
      	at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
      	at org.apache.kudu.shaded.org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:377)
      	at org.apache.kudu.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
      	at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
      	at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
      	at org.apache.kudu.shaded.org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525)
      	at org.apache.kudu.shaded.org.jboss.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:48)
      	at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.notifyHandlerException(DefaultChannelPipeline.java:658)
      	at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:566)
      	at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
      	at org.apache.kudu.shaded.org.jboss.netty.handler.timeout.ReadTimeoutHandler.messageReceived(ReadTimeoutHandler.java:184)
      	at org.apache.kudu.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
      	at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
      	at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
      	at org.apache.kudu.shaded.org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
      	at org.apache.kudu.shaded.org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:70)
      	at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
      	at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
      	at org.apache.kudu.shaded.org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
      	at org.apache.kudu.shaded.org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
      	at org.apache.kudu.shaded.org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
      	at org.apache.kudu.shaded.org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
      	at org.apache.kudu.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
      	at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
      	at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
      	at org.apache.kudu.shaded.org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
      	at org.apache.kudu.shaded.org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
      	at org.apache.kudu.shaded.org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
      	at org.apache.kudu.shaded.org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
      	at org.apache.kudu.shaded.org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
      	at org.apache.kudu.shaded.org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
      	at org.apache.kudu.shaded.org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
      	at org.apache.kudu.shaded.org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
      	at org.apache.kudu.shaded.org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      

        Issue Links

          Activity

          Hide
          danburkert Dan Burkert added a comment -

          This may be a duplicate of KUDU-2096.

          Show
          danburkert Dan Burkert added a comment - This may be a duplicate of KUDU-2096 .
          Hide
          tlipcon Todd Lipcon added a comment -

          I think this is specific to the Java client, and perhaps was fixed by KUDU-2103 already?

          Show
          tlipcon Todd Lipcon added a comment - I think this is specific to the Java client, and perhaps was fixed by KUDU-2103 already?
          Hide
          danburkert Dan Burkert added a comment -

          I observed this in 1.5, and KUDU-2103 is marked fixed as of 1.5.

          Show
          danburkert Dan Burkert added a comment - I observed this in 1.5, and KUDU-2103 is marked fixed as of 1.5.
          Hide
          tlipcon Todd Lipcon added a comment -

          ah... indeed. Attila Bukor any idea on this?

          Show
          tlipcon Todd Lipcon added a comment - ah... indeed. Attila Bukor any idea on this?
          Hide
          r1pp3rj4ck Attila Bukor added a comment -

          I'm looking into this.

          Show
          r1pp3rj4ck Attila Bukor added a comment - I'm looking into this.
          Hide
          danburkert Dan Burkert added a comment -

          Attila Bukor, Todd Lipcon and I discussed this today. The issue is that the Java client does do hostname canonicalization (due to KUDU-2103), but localhost canonicalizes to localhost. We could add a special-case for localhost, but using localhost as the master address is bad for another reason: it can't be resolved consistently across nodes. For kudu-spark-tools in particular, which does distributed processing, this is a big issue. So the conclusion is not to change the kudu-client, but to change the default master addr of the kudu-spark-tools job to the driver host's fqdn (https://gerrit.cloudera.org/#/c/8072/).

          HADOOP-9789 was raised as a possible solution, but it's not obviously secure, and requires more client-configuration that we are comfortable with, so for now the conclusion is not to use localhost as the master address in productionized applications.

          Show
          danburkert Dan Burkert added a comment - Attila Bukor , Todd Lipcon and I discussed this today. The issue is that the Java client does do hostname canonicalization (due to KUDU-2103 ), but localhost canonicalizes to localhost . We could add a special-case for localhost , but using localhost as the master address is bad for another reason: it can't be resolved consistently across nodes. For kudu-spark-tools in particular, which does distributed processing, this is a big issue. So the conclusion is not to change the kudu-client, but to change the default master addr of the kudu-spark-tools job to the driver host's fqdn ( https://gerrit.cloudera.org/#/c/8072/ ). HADOOP-9789 was raised as a possible solution, but it's not obviously secure, and requires more client-configuration that we are comfortable with, so for now the conclusion is not to use localhost as the master address in productionized applications.

            People

            • Assignee:
              r1pp3rj4ck Attila Bukor
              Reporter:
              danburkert Dan Burkert
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development