[FLUME-2792] Flume Kafka Kerberos Support

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Won't Fix
• Affects Version/s: 1.6.0, 1.5.2
• Component/s: Configuration, Docs, Sinks+Sources

Environment

HDP 2.3 fully kerberized including Kafka 0.8.2.2 + Flume 1.5.2 or Apache Flume 1.6 downloaded from apache.org

• Assignee: Unassigned
• Reporter: Hari Sekhon
• Created: 11/Sep/15 09:43
• Updated: 23/May/16 12:21
• Resolved: 17/May/16 15:48

Description

Following on from FLUME-2790 it appears as though Flume doesn't yet have support for Kafka + Kerberos as there are is no setting documented in the Flume 1.6.0 user guide under the Kafka source section to tell Flume to use plaintextsasl as the connection mechanism to Kafka and Kafka rejects unauthenticated plaintext mechanism:

15/09/10 16:51:22 INFO consumer.ConsumerFetcherManager: [ConsumerFetcherManager-1441903874830] Added fetcher for partitions ArrayBuffer()
15/09/10 16:51:22 WARN consumer.ConsumerFetcherManager$LeaderFinderThread: [flume_<custom_scrubbed>-1441903874763-abdc98ec-leader-finder-thread], Failed to find leader for Set([<custom_scrubbed>,0], [<custom_scrubbed>,1])
kafka.common.BrokerEndPointNotAvailableException: End point PLAINTEXT not found for broker 0
        at kafka.cluster.Broker.getBrokerEndPoint(Broker.scala:140)
        at kafka.utils.ZkUtils$$anonfun$getAllBrokerEndPointsForChannel$1.apply(ZkUtils.scala:124)
        at kafka.utils.ZkUtils$$anonfun$getAllBrokerEndPointsForChannel$1.apply(ZkUtils.scala:124)
        at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
        at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
        at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
        at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
        at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
        at scala.collection.AbstractTraversable.map(Traversable.scala:105)
        at kafka.utils.ZkUtils$.getAllBrokerEndPointsForChannel(ZkUtils.scala:124)
        at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66)
        at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)

Issue Links

• duplicates: FLUME-2820 Support New Kafka APIs
• relates to: FLUME-2819 Kafka libs are being bundled into Flume distro