Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-32760

Version Conflict in flink-sql-connector-hive for shaded.parquet prefix packages

    XMLWordPrintableJSON

Details

    Description

      Summary

      In https://issues.apache.org/jira/browse/FLINK-23074 it seems like shading parquet dependency from hive-exec is done. 

      But I think this is not enough and causing errors like below when I try to read parquet file using sql-gateway which requires both flink-parquet and flink-sql-connector-hive dependencies.

       

      Cause

      Parquet dependency not only includes org.apache.parquet but also shaded.parquet prefix dependencies. (ref)

      So we need to shade both.

      - flink-parquet depends on Parquet 1.12.3 with shaded Thrift 0.16.0 (prefix: shaded.parquet)

      - flink-sql-connector-hive depends on hive-exec 3.1.3 with Parquet 1.10.0 and shaded Thrift 0.9.3 (prefix: shaded.parquet)

      - Code compiled against Thrift 0.16.0 attempts to run against 0.9.3, causing the error.

      Proposed solution

      Adding new shading rule to flink-sql-connector-hive project.

      I have confirmed that adding this rule could resolve the above error.

      <relocation>
           <pattern>shaded.parquet</pattern>
           <shadedPattern>shaded.parquet.flink.hive.shaded</shadedPattern>
      </relocation>

       

      I would be happy to implement it if the proposal is accepted. Thanks

       

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              dongwoo.kim dongwoo.kim
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: