[DRILL-6739] Update Kafka libs to 2.0.0+ version

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.14.0
• Fix Version/s: 1.17.0
• Component/s: Storage - Kafka
• Assignee: Arina Ielchiieva
• Reporter: Vitalii Diravka
• Created: 11/Sep/18 09:51
• Updated: 20/Nov/19 11:45
• Resolved: 20/Nov/19 11:45

Description

The current version of Kafka libs is 0.11.0.1
The last version is 2.0.0 (September 2018) https://mvnrepository.com/artifact/org.apache.kafka/kafka-clients

Looks like the only changes which should be done are:

• replacing serverConfig() method with staticServerConfig() in Drill EmbeddedKafkaCluster class
• Replacing deprecated AdminUtils with kafka.zk.AdminZkClient https://github.com/apache/kafka/blob/3cdc78e6bb1f83973a14ce1550fe3874f7348b05/core/src/main/scala/kafka/admin/AdminUtils.scala#L35
  https://issues.apache.org/jira/browse/KAFKA-6545

The initial work: https://github.com/vdiravka/drill/commits/DRILL-6739

Issue Links

• is part of: DRILL-7388 Apache Drill Kafka Storage module fails to return results for partitions containing single offset record
• relates to: KAFKA-5103 Refactor AdminUtils to use zkUtils methods instad of zkUtils.zkClient
• relates to: KAFKA-1420 Replace AdminUtils.createOrUpdateTopicPartitionAssignmentPathInZK with TestUtils.createTopic in unit tests
• relates to: KAFKA-6545 AdminUtils.fetchTopicMetadataFromZk is not available in 1.0.0
• links to: DRILL-6739: Initial work