Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-13945

Instructions for building flink-shaded against vendor repository don't work

    XMLWordPrintableJSON

    Details

      Description

      According to documentation, to build Flink against a vendor specific Hadoop version it is necessary to build flink-shaded against this version first : 

      mvn clean install -DskipTests -Pvendor-repos -Dhadoop.version=<hadoop_version>
      

      vendor-repos profile has to be activated to include Hadoop vendors repositories.
      But Maven cannot find expected Hadoop dependencies and returns an error because vendor-repos profile isn't defined in flink-shaded.

      Example using flink-shaded 8.0 and HDP 2.6.5 Hadoop version :

      mvn clean install -DskipTests -Pvendor-repos -Dhadoop.version=2.7.3.2.6.5.0-292
      
      [INFO] ---------------< org.apache.flink:flink-shaded-hadoop-2 >---------------
      [INFO] Building flink-shaded-hadoop-2 2.7.3.2.6.5.0-292-8.0             [10/11]
      [INFO] --------------------------------[ jar ]---------------------------------
      [WARNING] The POM for org.apache.hadoop:hadoop-common:jar:2.7.3.2.6.5.0-292 is missing, no dependency information available
      [WARNING] The POM for org.apache.hadoop:hadoop-hdfs:jar:2.7.3.2.6.5.0-292 is missing, no dependency information available
      [WARNING] The POM for org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.7.3.2.6.5.0-292 is missing, no dependency information available
      [WARNING] The POM for org.apache.hadoop:hadoop-yarn-client:jar:2.7.3.2.6.5.0-292 is missing, no dependency information available
      [WARNING] The POM for org.apache.hadoop:hadoop-yarn-common:jar:2.7.3.2.6.5.0-292 is missing, no dependency information available
      [INFO] ------------------------------------------------------------------------
      [INFO] Reactor Summary:
      [INFO]
      [INFO] flink-shaded 8.0 ................................... SUCCESS [  2.122 s]
      [INFO] flink-shaded-force-shading 8.0 ..................... SUCCESS [  0.607 s]
      [INFO] flink-shaded-asm-7 7.1-8.0 ......................... SUCCESS [  0.667 s]
      [INFO] flink-shaded-guava-18 18.0-8.0 ..................... SUCCESS [  1.452 s]
      [INFO] flink-shaded-netty-4 4.1.39.Final-8.0 .............. SUCCESS [  4.597 s]
      [INFO] flink-shaded-netty-tcnative-dynamic 2.0.25.Final-8.0 SUCCESS [  0.620 s]
      [INFO] flink-shaded-jackson-parent 2.9.8-8.0 .............. SUCCESS [  0.018 s]
      [INFO] flink-shaded-jackson-2 2.9.8-8.0 ................... SUCCESS [  0.914 s]
      [INFO] flink-shaded-jackson-module-jsonSchema-2 2.9.8-8.0 . SUCCESS [  0.627 s]
      [INFO] flink-shaded-hadoop-2 2.7.3.2.6.5.0-292-8.0 ........ FAILURE [  0.047 s]
      [INFO] flink-shaded-hadoop-2-uber 2.7.3.2.6.5.0-292-8.0 ... SKIPPED
      [INFO] ------------------------------------------------------------------------
      [INFO] BUILD FAILURE
      [INFO] ------------------------------------------------------------------------
      [INFO] Total time:  11.947 s
      [INFO] Finished at: 2019-09-03T16:52:59+02:00
      [INFO] ------------------------------------------------------------------------
      [WARNING] The requested profile "vendor-repos" could not be activated because it does not exist.
      [ERROR] Failed to execute goal on project flink-shaded-hadoop-2: Could not resolve dependencies for project org.apache.flink:flink-shaded-hadoop-2:jar:2.7.3.2.6.5.0-292-8.0: The following artifacts could not be resolved: org.apache.hadoop:hadoop-common:jar:2.7.3.2.6.5.0-292, org.apache.hadoop:hadoop-hdfs:jar:2.7.3.2.6.5.0-292, org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.7.3.2.6.5.0-292, org.apache.hadoop:hadoop-yarn-client:jar:2.7.3.2.6.5.0-292, org.apache.hadoop:hadoop-yarn-common:jar:2.7.3.2.6.5.0-292: Failure to find org.apache.hadoop:hadoop-common:jar:2.7.3.2.6.5.0-292 in https://repo.maven.apache.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced -> [Help 1]
      [ERROR]
      [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
      [ERROR] Re-run Maven using the -X switch to enable full debug logging.
      [ERROR]
      [ERROR] For more information about the errors and possible solutions, please read the following articles:
      [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
      [ERROR]
      [ERROR] After correcting the problems, you can resume the build with the command
      [ERROR]   mvn <goals> -rf :flink-shaded-hadoop-2
      

      vendor-repos profile exists in Flink pom.xml file : https://github.com/apache/flink/blob/3079d11913f153ec40c75afb5356fd3be1a1e550/pom.xml#L1037

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                chesnay Chesnay Schepler
                Reporter:
                Elise Ramé Elise Ramé
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 20m
                  20m