[FLINK-32760] Version Conflict in flink-sql-connector-hive for shaded.parquet prefix packages

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.17.1
• Fix Version/s: 1.18.0, 1.16.3, 1.17.2
• Component/s: Connectors / Hive
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: dongwoo.kim
• Created: 05/Aug/23 06:04
• Updated: 25/Aug/23 01:06
• Resolved: 25/Aug/23 01:06

Description

Summary

In https://issues.apache.org/jira/browse/FLINK-23074 it seems like shading parquet dependency from hive-exec is done. 

But I think this is not enough and causing errors like below when I try to read parquet file using sql-gateway which requires both flink-parquet and flink-sql-connector-hive dependencies.

 

Cause

Parquet dependency not only includes org.apache.parquet but also shaded.parquet prefix dependencies. (ref)

So we need to shade both.

- flink-parquet depends on Parquet 1.12.3 with shaded Thrift 0.16.0 (prefix: shaded.parquet)

- flink-sql-connector-hive depends on hive-exec 3.1.3 with Parquet 1.10.0 and shaded Thrift 0.9.3 (prefix: shaded.parquet)

- Code compiled against Thrift 0.16.0 attempts to run against 0.9.3, causing the error.

Proposed solution

Adding new shading rule to flink-sql-connector-hive project.

I have confirmed that adding this rule could resolve the above error.

<relocation>
     <pattern>shaded.parquet</pattern>
     <shadedPattern>shaded.parquet.flink.hive.shaded</shadedPattern>
</relocation>

 

I would be happy to implement it if the proposal is accepted. Thanks

 

Issue Links

• links to: GitHub Pull Request #23166