Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-3441

Revisit quickstarts exclusion policy



    • Improvement
    • Status: Reopened
    • Not a Priority
    • Resolution: Unresolved
    • 1.0.0
    • None
    • Quickstarts
    • None


      Flink provides quickstart projects to quickly set up a Flink project. In order to decrease the size of the resulting jar, the predefined pom.xml file contains a list of excluded dependencies.

      The problem with this approach is two-fold: First of all, the list seems to be rather arbitrary. Why excluding org.apache.commons:commons-math but not org.apache.commons:commons-math3? It seems as if this list would have to be kept up to date with every changing dependency of Flink.

      Secondly and the more severe problem is the following: The exclusions assume that Flink always provides a compatible version for an excluded dependency. However, since the exclusions are not bound to a specific version, this won't necessarily be true. For example, snappy-java is excluded from the user job jar, because Flink with Hadoop 2.3.0 comes with snappy-java:1.0.5. However, the kafka-clients:0.9.0 which comes with the flink-connector-kafka-0.9 dependency has a transitive dependency on snappy-java: Consequently, this dependency will be excluded from the user job jar per default. If version is not compatible to snappy-java:1.0.5, then the program will crash at runtime. This will be very hard to understand for the user because it is a very subtle bug since the exclusions also affect the transitive dependencies.




            Unassigned Unassigned
            trohrmann Till Rohrmann
            0 Vote for this issue
            4 Start watching this issue