[FLINK-1065] "flink-streaming-core" build fails with Hadoop 2.4.0 dependencies

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.7.0-incubating
• Fix Version/s: 0.7.0-incubating
• Assignee: Márton Balassi
• Reporter: Robert Metzger
• Created: 26/Aug/14 10:21
• Updated: 28/Feb/19 11:16
• Resolved: 30/Aug/14 13:50

Description

Hi,

building Flink with "mvn clean package -Dhadoop.profile=2 -DskipTests -Dhadoop-two.version=2.4.0"

results in the following error

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project flink-streaming-core: Compilation failure: Compilation failure:
[ERROR] /home/robert/incubator-flink/flink-addons/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/api/invokable/operator/BatchReduceInvokable.java:[22,36] package org.apache.commons.math.util does not exist
[ERROR] /home/robert/incubator-flink/flink-addons/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/api/invokable/operator/WindowReduceInvokable.java:[22,36] package org.apache.commons.math.util does not exist
[ERROR] /home/robert/incubator-flink/flink-addons/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/api/invokable/operator/BatchReduceInvokable.java:[39,36] cannot find symbol
[ERROR] symbol:   variable MathUtils
[ERROR] location: class org.apache.flink.streaming.api.invokable.operator.BatchReduceInvokable<IN,OUT>
[ERROR] /home/robert/incubator-flink/flink-addons/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/api/invokable/operator/BatchReduceInvokable.java:[48,36] cannot find symbol
[ERROR] symbol:   variable MathUtils
[ERROR] location: class org.apache.flink.streaming.api.invokable.operator.BatchReduceInvokable<IN,OUT>
[ERROR] /home/robert/incubator-flink/flink-addons/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/api/invokable/operator/WindowReduceInvokable.java:[40,36] cannot find symbol
[ERROR] symbol:   variable MathUtils
[ERROR] location: class org.apache.flink.streaming.api.invokable.operator.WindowReduceInvokable<IN,OUT>
[ERROR] /home/robert/incubator-flink/flink-addons/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/api/invokable/operator/WindowReduceInvokable.java:[49,36] cannot find symbol
[ERROR] symbol:   variable MathUtils
[ERROR] location: class org.apache.flink.streaming.api.invokable.operator.WindowReduceInvokable<IN,OUT>
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :flink-streaming-core

I think older Hadoop versions (its working with Hadoop 2.2.0) are pulling in transitive dependencies that are not present with Hadoop 2.4.0 anymore.