Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-3335

Fix Mahout build after upgrading Spark to 2.4.5

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.5.0
    • Component/s: build
    • Labels:
      None

      Description

      After BIGTOP-3165, building Mahout came to fail as follows, because Scala 2.10 support was removed from Spark 2.3.

      $ ./gradlew mahout-pkg
      
      (snip)
      
      [INFO] ----------------< org.apache.mahout:mahout-spark_2.10 >-----------------
      [INFO] Building Mahout Spark bindings 0.13.0                             [9/11]
      [INFO] --------------------------------[ jar ]---------------------------------
      [WARNING] The POM for org.apache.spark:spark-core_2.10:jar:2.4.5 is missing, no dependency information available
      [WARNING] The POM for org.apache.spark:spark-mllib_2.10:jar:2.4.5 is missing, no dependency information available
      [INFO] ------------------------------------------------------------------------
      [INFO] Reactor Summary for Apache Mahout 0.13.0:
      [INFO] 
      [INFO] Mahout Build Tools ................................. SUCCESS [  2.564 s]
      [INFO] Apache Mahout ...................................... SUCCESS [  0.047 s]
      [INFO] Mahout Math ........................................ SUCCESS [ 11.193 s]
      [INFO] Mahout HDFS ........................................ SUCCESS [  3.203 s]
      [INFO] Mahout Map-Reduce .................................. SUCCESS [ 14.859 s]
      [INFO] Mahout Integration ................................. SUCCESS [  2.582 s]
      [INFO] Mahout Examples .................................... SUCCESS [ 14.055 s]
      [INFO] Mahout Math Scala bindings ......................... SUCCESS [ 30.022 s]
      [INFO] Mahout Spark bindings .............................. FAILURE [  0.029 s]
      [INFO] Mahout Release Package ............................. SKIPPED
      [INFO] Mahout H2O backend ................................. SKIPPED
      [INFO] ------------------------------------------------------------------------
      [INFO] BUILD FAILURE
      [INFO] ------------------------------------------------------------------------
      [INFO] Total time:  01:18 min
      [INFO] Finished at: 2020-04-03T17:07:38+09:00
      [INFO] ------------------------------------------------------------------------
      [ERROR] Failed to execute goal on project mahout-spark_2.10: Could not resolve dependencies for project org.apache.mahout:mahout-spark_2.10:jar:0.13.0: The following artifacts could not be resolved: org.apache.spark:spark-core_2.10:jar:2.4.5, org.apache.spark:spark-mllib_2.10:jar:2.4.5: Failure to find org.apache.spark:spark-core_2.10:jar:2.4.5 in https://repository.cloudera.com/artifactory/cloudera-repos/ was cached in the local repository, resolution will not be reattempted until the update interval of cloudera has elapsed or updates are forced -> [Help 1]
      [ERROR] 
      [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
      [ERROR] Re-run Maven using the -X switch to enable full debug logging.
      [ERROR] 
      [ERROR] For more information about the errors and possible solutions, please read the following articles:
      [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
      [ERROR] 
      [ERROR] After correcting the problems, you can resume the build with the command
      [ERROR]   mvn <args> -rf :mahout-spark_2.10
      debian/rules:30: recipe for target 'override_dh_auto_build' failed
      make[1]: *** [override_dh_auto_build] Error 1
      make[1]: Leaving directory '/home/sekikn/repos/bigtop/output/mahout/mahout-0.13.0'
      debian/rules:27: recipe for target 'build' failed
      make: *** [build] Error 2
      dpkg-buildpackage: error: debian/rules build subprocess returned exit status 2
      debuild: fatal error at line 1152:
      dpkg-buildpackage -rfakeroot -us -uc -ui -b
       failed
      
      > Task :mahout-deb FAILED
      
      FAILURE: Build failed with an exception.
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                sekikn Kengo Seki
                Reporter:
                sekikn Kengo Seki
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 50m
                  50m