[HIVE-20639] Add ability to Write Data from Hive Table/Query to Kafka Topic

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.0.0-alpha-1
• Component/s: kafka integration
• Assignee: Slim Bouguerra
• Reporter: Slim Bouguerra
• Created: 26/Sep/18 17:06
• Updated: 17/Nov/22 08:55
• Resolved: 13/Oct/18 17:49

Description

This patch adds multiple record writers to allow Hive user writing data directly to a Kafka Topic.
The end goal is have a writer that provides multiple write semantics modes.

• • A BEST_EFFORT where all the records will be delivered with no guarantee or reties eg best effort mode. (Done)-
• B At_least_once, each record will be delivered with retries from the Kafka Producer and Hive Write Task. (Done)
• C Exactly_once , Writer will be using Kafka Transaction API to ensure that each record is delivered once. (not so, still need more discussions)

In addition to the new feature i have refactored the existing code to make it more readable.

Issue Links

• links to: Design Doc
• links to: Github PR
• links to: RB Link

Sub-Tasks

• Address some of the review comments plus Kerberos support: Closed