[STORM-2560] Storm-Kafka on CDH 5.11 with kerberos security enabled.

• Type: Question
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.1.0
• Component/s: storm-kafka
• Labels: security
• Assignee: Unassigned
• Reporter: Niraj Parmar
• Created: 20/Jun/17 09:33
• Updated: 25/Oct/17 09:19

Description

Hi,

I have installed Apache Storm 1.1.0 manually on CDH 5.11 cluster. This cluster is secured with kerberos.
I have storm sample written which ingest data from kafka topic and inserts into HDFS directory in real time. So, this sample uses storm-kafka as well as storm-hdfs.
When I run the storm topology it gives the following error in kafka-spout.

2017-06-18 22:29:31.297 o.a.z.ClientCnxn Thread-14-kafka-spout-executor[5 5]-SendThread(localhost:2181) [INFO] Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)

2017-06-18 22:29:31.571 k.c.SimpleConsumer Thread-14-kafka-spout-executor[5 5] [INFO] Reconnect due to error:
java.nio.channels.ClosedChannelException: null
at kafka.network.BlockingChannel.send(BlockingChannel.scala:110) ~[stormjar.jar:?]
at kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.scala:85) [stormjar.jar:?]
at kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:83) [stormjar.jar:?]
at kafka.consumer.SimpleConsumer.getOffsetsBefore(SimpleConsumer.scala:149) [stormjar.jar:?]
at kafka.javaapi.consumer.SimpleConsumer.getOffsetsBefore(SimpleConsumer.scala:79) [stormjar.jar:?]
at org.apache.storm.kafka.KafkaUtils.getOffset(KafkaUtils.java:75) [stormjar.jar:?]
at org.apache.storm.kafka.KafkaUtils.getOffset(KafkaUtils.java:65) [stormjar.jar:?]
at org.apache.storm.kafka.PartitionManager.<init>(PartitionManager.java:94) [stormjar.jar:?]
at org.apache.storm.kafka.ZkCoordinator.refresh(ZkCoordinator.java:98) [stormjar.jar:?]
at org.apache.storm.kafka.ZkCoordinator.getMyManagedPartitions(ZkCoordinator.java:69) [stormjar.jar:?]
at org.apache.storm.kafka.KafkaSpout.nextTuple(KafkaSpout.java:129) [stormjar.jar:?]
at org.apache.storm.daemon.executor$fn_4976$fn4991$fn_5022.invoke(executor.clj:644) [storm-core-1.1.0.jar:1.1.0]
at org.apache.storm.util$async_loop$fn__557.invoke(util.clj:484) [storm-core-1.1.0.jar:1.1.0]

Kafka version: 2.1.1-1.2.1.1.p0.18

There is no storm-kafka*.jat present in - "/usr/local/storm"
But this sample was workin fine before kerberizing the cluster, even in this case.

I have tried the same example on Hortonworks and after adding the below code to set security protcol, the topology runs fine:
spoutConfig.securityProtocol = "SASL_PLAINTEXT";
After Adding above code in case of cloudera it gives error: "Symbol not found"

Please let me know if you nedd any other information...
Thanks in advance..