Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-30400

Stop bundling connector-base in externalized connectors

Details

    Description

      Check that none of the externalized connectors bundle connector-base; if so remove the bundling and schedule a new minor release.

      Bundling this module is highly problematic w.r.t. binary compatibility, since bundled classes may rely on internal APIs.

      Attachments

        Activity

          ruanhang1993 Ruan Hang added a comment -

          Hi, all. I would like to help to check. Thanks.

          ruanhang1993 Ruan Hang added a comment - Hi, all. I would like to help to check. Thanks.
          martijnvisser Martijn Visser added a comment -

          chesnay How would this work for SQL FAT jars? IIUC they actually need connector-base

          martijnvisser Martijn Visser added a comment - chesnay How would this work for SQL FAT jars? IIUC they actually need connector-base
          dannycranmer Danny Cranmer added a comment - AWS: 01f112bd5a69f95cd5d2a4bc7e08d1ba9a81d56a
          dannycranmer Danny Cranmer added a comment -

          > Chesnay Schepler How would this work for SQL FAT jars? IIUC they actually need connector-base

          martijnvisser flink-connector-base is included in Flink dist, so SQL jars are fine; it will already be on the classpath.

          dannycranmer Danny Cranmer added a comment - > Chesnay Schepler How would this work for SQL FAT jars? IIUC they actually need connector-base martijnvisser flink-connector-base is included in Flink dist , so SQL jars are fine; it will already be on the classpath.
          martijnvisser Martijn Visser added a comment -

          dannycranmer Thanks

          martijnvisser Martijn Visser added a comment - dannycranmer Thanks
          martijnvisser Martijn Visser added a comment -

          Fixed in apache/flink-connector-kafka:main 37cbb83f55e9d6f0b8dc35bb8da867086dfa4d9e

          martijnvisser Martijn Visser added a comment - Fixed in apache/flink-connector-kafka:main 37cbb83f55e9d6f0b8dc35bb8da867086dfa4d9e
          martijnvisser Martijn Visser added a comment -

          Fixed in apache/flink-connector-rabbitmq:

          main: fb0566abb45bf545668c688311253056eb7b4192
          v3.0: bb9936513803c895b3e2ac7cb9af525bbe685f4c

          martijnvisser Martijn Visser added a comment - Fixed in apache/flink-connector-rabbitmq: main: fb0566abb45bf545668c688311253056eb7b4192 v3.0: bb9936513803c895b3e2ac7cb9af525bbe685f4c
          leonard Leonard Xu added a comment -

          Hey ruanhang1993, recently we meet some issue when release connector like kafka[1] ,we need pick batch from connector main branch to dev branch, would you like to push things forward ?
          [1] https://lists.apache.org/thread/qfb383vl321gcyv10fw6wb6jp9xjzrx4

          leonard Leonard Xu added a comment - Hey ruanhang1993 , recently we meet some issue when release connector like kafka [1] ,we need pick batch from connector main branch to dev branch, would you like to push things forward ? [1] https://lists.apache.org/thread/qfb383vl321gcyv10fw6wb6jp9xjzrx4

          fixed for apache/flink-connector-kafka:v3.0 as well:
          44e49e777f78bd3d13ffff843f0711f617d373e4

          tzulitai Tzu-Li (Gordon) Tai added a comment - fixed for apache/flink-connector-kafka:v3.0 as well: 44e49e777f78bd3d13ffff843f0711f617d373e4
          rmetzger Robert Metzger added a comment -

          The problem of marking flink-connector-base provided for a connector is that classes like the DeliveryGuarantee are missing when developing locally.
          For example the KafkaSink example includes the DeliveryGuarantee, which is not pulled in by my IDE because it a transitive provided dependency.

          We could also update the connector docs and list the flink-connector-base as well?

          rmetzger Robert Metzger added a comment - The problem of marking flink-connector-base provided for a connector is that classes like the DeliveryGuarantee are missing when developing locally. For example the KafkaSink example includes the DeliveryGuarantee , which is not pulled in by my IDE because it a transitive provided dependency. We could also update the connector docs and list the flink-connector-base as well?
          ruanhang1993 Ruan Hang added a comment -

          rmetzger ,Thanks for the suggestion.

          I will write this content in the part of docs/connectors/datastream/overview.

          ruanhang1993 Ruan Hang added a comment - rmetzger ,Thanks for the suggestion. I will write this content in the part of docs/connectors/datastream/overview.
          rmetzger Robert Metzger added a comment -

          Thanks. I'm happy to review.
          I wonder if martijnvisser or chesnay have any better ideas how to deal with the issue.

          rmetzger Robert Metzger added a comment - Thanks. I'm happy to review. I wonder if martijnvisser or chesnay have any better ideas how to deal with the issue.
          dannycranmer Danny Cranmer added a comment -

          I agree this is frustrating. It is a similar problem with flink-streaming-java and flink-clients that I am sure many users stumble on. Personally I create an "intellij" Maven profile that I enable in the IDE which includes these dependencies. It would be ideal if we could provide a Flink dev plugin or similar that sets these things up.

          dannycranmer Danny Cranmer added a comment - I agree this is frustrating. It is a similar problem with flink-streaming-java and flink-clients that I am sure many users stumble on. Personally I create an "intellij" Maven profile that I enable in the IDE which includes these dependencies. It would be ideal if we could provide a Flink dev plugin or similar that sets these things up.

          People

            ruanhang1993 Ruan Hang
            chesnay Chesnay Schepler
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: