[HIVE-27402] Refactor Kafka credential handling into kafka-handler module

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.0.0-beta-1
• Component/s: HiveServer2, kafka integration
• Labels: pull-request-available
• Assignee: Stamatis Zampetakis
• Reporter: Stamatis Zampetakis
• Created: 02/Jun/23 11:14
• Updated: 15/Aug/23 06:33
• Resolved: 08/Jun/23 07:50

Description

Currently the code for handling credentials and authentication for the Kafka storage handler is spread across hive-exec (https://github.com/apache/hive/blob/7cd3107a76d633ef5fae2ffb8ec16953ac968092/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java#L302) and kafka-handler (https://github.com/apache/hive/blob/7cd3107a76d633ef5fae2ffb8ec16953ac968092/kafka-handler/src/java/org/apache/hadoop/hive/kafka/KafkaUtils.java#L71) modules. This leads to unexpected dependencies to kafka (e.g., kafka-client) in hive-exec module and limits sharing opportunities of credential utilities present in KafkaUtil class.

The goal of this refactoring is threefold:

• centralize Kafka credential handling (in one module) opening the road to more code reuse and better encapsulation;
• remove kafka dependencies (e.g., kafka-client) from hive-exec module;
• facilitate unit testing and product stability.

Issue Links

• is related to: HIVE-27475 Downgrade Kafka version to 2.5.0 to stabilise Kafka handler
• links to: GitHub Pull Request #4385