[HIVE-27475] Downgrade Kafka version to 2.5.0 to stabilise Kafka handler

• Type: Task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0-alpha-2
• Fix Version/s: 4.0.0-beta-1
• Component/s: kafka integration
• Labels: pull-request-available
• Assignee: Akshat Mathur
• Reporter: Akshat Mathur
• Created: 30/Jun/23 05:58
• Updated: 15/Aug/23 06:33
• Resolved: 03/Jul/23 12:33

Description

It was needed to shade a version of zstd-jni that is compatible with parquet in hive-ql after upgrading parquet version dowstream, otherwise Tez couldn't work with parquet and zstd encryption. Parquet and kafka-client both use zstd-jni. In hive-ql, zstd-jni is coming as a transitive dependency from kafka-client. The zstd-jni version in kafka-client 2.5.0 was not compatible with parquet, so in PR-4082 kafka-client was upgraded to latest version which used zstd-jni version close to the version in parquet.

While the upgrade fixed the zstd-ini issue, It introduced multiple compatibility issue in the Kafka-handler and due to lack to test cases and disabled tests these issues went unidentified.

With the recent refactor in HIVE-27402, Kafka-client dependency was moved out of hive-ql, Which fixed the original zstd-ini issue.(confirmed by difin)

Hence , It makes more sense to downgrade kafka versions back to 2.5.0, stabalize kafka-handler, enable test and then upgrade.

Ref to discussion can be found here: #4436

Issue Links

• is related to: HIVE-27502 Fix flaky test HiveKafkaProducerTest
• relates to: HIVE-27105 Querying parquet table with zstd encryption is failing
• relates to: HIVE-27402 Refactor Kafka credential handling into kafka-handler module
• links to: GitHub Pull Request #4460