Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-3334

Fix Oozie build after upgrading Spark to 2.4.5

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.5.0
    • Component/s: build
    • Labels:
      None

      Description

      After BIGTOP-3165, building Oozie came to fail as follows, because Scala 2.10 support was removed from Spark 2.3.

      $ ./gradlew oozie-pkg
      
      (snip)
      
      [INFO] ------------------< org.apache.oozie:oozie-examples >-------------------
      [INFO] Building Apache Oozie Examples 4.3.0                             [15/23]
      [INFO] --------------------------------[ jar ]---------------------------------
      Downloading from central: https://repo1.maven.org/maven2/org/apache/spark/spark-core_2.10/2.4.5/spark-core_2.10-2.4.5.pom
      Downloading from apache.snapshots.repo: https://repository.apache.org/content/groups/snapshots/org/apache/spark/spark-core_2.10/2.4.5/spark-core_2.10-2.4.5.pom
      Downloading from redhat: https://maven.repository.redhat.com/ga/org/apache/spark/spark-core_2.10/2.4.5/spark-core_2.10-2.4.5.pom
      [WARNING] The POM for org.apache.spark:spark-core_2.10:jar:2.4.5 is missing, no dependency information available
      Downloading from central: https://repo1.maven.org/maven2/org/apache/spark/spark-core_2.10/2.4.5/spark-core_2.10-2.4.5.jar
      Downloading from apache.snapshots.repo: https://repository.apache.org/content/groups/snapshots/org/apache/spark/spark-core_2.10/2.4.5/spark-core_2.10-2.4.5.jar
      Downloading from redhat: https://maven.repository.redhat.com/ga/org/apache/spark/spark-core_2.10/2.4.5/spark-core_2.10-2.4.5.jar
      [INFO] ------------------------------------------------------------------------
      [INFO] Reactor Summary:
      [INFO] 
      [INFO] Apache Oozie Main 4.3.0 ............................ SUCCESS [  1.526 s]
      [INFO] Apache Oozie Hadoop Utils hadoop-2-4.3.0 hadoop-2-4.3.0 SUCCESS [  2.526 s]
      [INFO] Apache Oozie Hadoop Distcp hadoop-2-4.3.0 hadoop-2-4.3.0 SUCCESS [  0.121 s]
      [INFO] Apache Oozie Hadoop Auth hadoop-2-4.3.0 Test hadoop-2-4.3.0 SUCCESS [  0.275 s]
      [INFO] Apache Oozie Hadoop Libs 4.3.0 ..................... SUCCESS [  0.020 s]
      [INFO] Apache Oozie Client 4.3.0 .......................... SUCCESS [  9.910 s]
      [INFO] Apache Oozie Share Lib Oozie 4.3.0 ................. SUCCESS [  2.319 s]
      [INFO] Apache Oozie Share Lib HCatalog 4.3.0 .............. SUCCESS [  7.790 s]
      [INFO] Apache Oozie Share Lib Distcp 4.3.0 ................ SUCCESS [  0.645 s]
      [INFO] Apache Oozie Core 4.3.0 ............................ SUCCESS [ 39.494 s]
      [INFO] Apache Oozie Share Lib Streaming 4.3.0 ............. SUCCESS [  4.785 s]
      [INFO] Apache Oozie Share Lib Hive 4.3.0 .................. SUCCESS [  8.152 s]
      [INFO] Apache Oozie Share Lib Hive 2 4.3.0 ................ SUCCESS [  6.853 s]
      [INFO] Apache Oozie Share Lib Sqoop 4.3.0 ................. SUCCESS [  3.060 s]
      [INFO] Apache Oozie Examples 4.3.0 ........................ FAILURE [  5.389 s]
      [INFO] Apache Oozie Share Lib Spark 4.3.0 ................. SKIPPED
      [INFO] Apache Oozie Share Lib 4.3.0 ....................... SKIPPED
      [INFO] Apache Oozie Docs 4.3.0 ............................ SKIPPED
      [INFO] Apache Oozie WebApp 4.3.0 .......................... SKIPPED
      [INFO] Apache Oozie Tools 4.3.0 ........................... SKIPPED
      [INFO] Apache Oozie MiniOozie 4.3.0 ....................... SKIPPED
      [INFO] Apache Oozie Distro 4.3.0 .......................... SKIPPED
      [INFO] Apache Oozie ZooKeeper Security Tests 4.3.0 ........ SKIPPED
      [INFO] ------------------------------------------------------------------------
      [INFO] BUILD FAILURE
      [INFO] ------------------------------------------------------------------------
      [INFO] Total time:  01:33 min
      [INFO] Finished at: 2020-04-03T11:48:09+09:00
      [INFO] ------------------------------------------------------------------------
      [ERROR] Failed to execute goal on project oozie-examples: Could not resolve dependencies for project org.apache.oozie:oozie-examples:jar:4.3.0: Could not find artifact org.apache.spark:spark-core_2.10:jar:2.4.5 in central (https://repo1.maven.org/maven2) -> [Help 1]
      [ERROR] 
      [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
      [ERROR] Re-run Maven using the -X switch to enable full debug logging.
      [ERROR] 
      [ERROR] For more information about the errors and possible solutions, please read the following articles:
      [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
      [ERROR] 
      [ERROR] After correcting the problems, you can resume the build with the command
      [ERROR]   mvn <args> -rf :oozie-examples
      
      ERROR, Oozie distro creation failed
      
      + MKDISTRO_RESULT=255
      + '[' 255 '!=' 0 ']'
      + echo 'ERROR: mkdistro.sh failed with error 255'
      ERROR: mkdistro.sh failed with error 255
      + echo 'Build files from failed build: /home/sekikn/repos/bigtop/output/oozie/oozie-4.3.0'
      Build files from failed build: /home/sekikn/repos/bigtop/output/oozie/oozie-4.3.0
      + exit 255
      debian/rules:33: recipe for target 'override_dh_auto_build' failed
      make[1]: *** [override_dh_auto_build] Error 255
      make[1]: Leaving directory '/home/sekikn/repos/bigtop/output/oozie/oozie-4.3.0'
      debian/rules:27: recipe for target 'build' failed
      make: *** [build] Error 2
      dpkg-buildpackage: error: debian/rules build subprocess returned exit status 2
      debuild: fatal error at line 1152:
      dpkg-buildpackage -rfakeroot -us -uc -ui -b
       failed
      
      > Task :oozie-deb FAILED
      
      FAILURE: Build failed with an exception.
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                sekikn Kengo Seki
                Reporter:
                sekikn Kengo Seki
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 40m
                  40m