Details
-
Improvement
-
Status: Reopened
-
Not a Priority
-
Resolution: Unresolved
-
1.0.0
-
None
-
None
Description
Flink provides quickstart projects to quickly set up a Flink project. In order to decrease the size of the resulting jar, the predefined pom.xml file contains a list of excluded dependencies.
The problem with this approach is two-fold: First of all, the list seems to be rather arbitrary. Why excluding org.apache.commons:commons-math but not org.apache.commons:commons-math3? It seems as if this list would have to be kept up to date with every changing dependency of Flink.
Secondly and the more severe problem is the following: The exclusions assume that Flink always provides a compatible version for an excluded dependency. However, since the exclusions are not bound to a specific version, this won't necessarily be true. For example, snappy-java is excluded from the user job jar, because Flink with Hadoop 2.3.0 comes with snappy-java:1.0.5. However, the kafka-clients:0.9.0 which comes with the flink-connector-kafka-0.9 dependency has a transitive dependency on snappy-java:1.1.1.7. Consequently, this dependency will be excluded from the user job jar per default. If version 1.1.1.7 is not compatible to snappy-java:1.0.5, then the program will crash at runtime. This will be very hard to understand for the user because it is a very subtle bug since the exclusions also affect the transitive dependencies.