Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-3336

Fix Livy build on CentOS and Fedora

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.5.0
    • Component/s: build
    • Labels:
      None

      Description

      On CentOS 7, building Livy fails with the following error, due to the lack of the wheel package.

      centos-7
      $ ./gradlew livy-pkg
      
      (snip)
      
      [INFO] --- exec-maven-plugin:1.2.1:exec (python-api install) @ livy-python-api ---
      /usr/lib/python2.7/site-packages/pkg_resources/py2_warn.py:22: UserWarning: Setuptools will stop working on Python 2
      ************************************************************
      You are running Setuptools on Python 2, which is no longer
      supported and
      >>> SETUPTOOLS WILL STOP WORKING <<<
      in a subsequent release (no sooner than 2020-04-20).
      Please ensure you are installing
      Setuptools using pip 9.x or later or pin to `setuptools<45`
      in your environment.
      If you have done those things and are still encountering
      this message, please comment in
      https://github.com/pypa/setuptools/issues/1458
      about the steps that led to this unsupported combination.
      ************************************************************
        sys.version_info < (3,) and warnings.warn(pre + "*" * 60 + msg + "*" * 60)
      WARNING: The wheel package is not available.
      ERROR: 'pip wheel' requires the 'wheel' package. To fix this, run: pip install wheel
      Traceback (most recent call last):
        File "setup.py", line 57, in <module>
          tests_require=['pytest']
        File "/usr/lib/python2.7/site-packages/setuptools/__init__.py", line 143, in setup
          _install_setup_requires(attrs)
        File "/usr/lib/python2.7/site-packages/setuptools/__init__.py", line 138, in _install_setup_requires
          dist.fetch_build_eggs(dist.setup_requires)
        File "/usr/lib/python2.7/site-packages/setuptools/dist.py", line 687, in fetch_build_eggs
          replace_conflicting=True,
        File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 783, in resolve
          replace_conflicting=replace_conflicting
        File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1066, in best_match
          return self.obtain(req, installer)
        File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1078, in obtain
          return installer(requirement)
        File "/usr/lib/python2.7/site-packages/setuptools/dist.py", line 743, in fetch_build_egg
          return fetch_build_egg(self, req)
        File "/usr/lib/python2.7/site-packages/setuptools/installer.py", line 130, in fetch_build_egg
          raise DistutilsError(str(e))
      distutils.errors.DistutilsError: Command '['/usr/bin/python', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmpwZ8oio', '--quiet', 'pytest-runner']' returned non-zero exit status 1
      [INFO] ------------------------------------------------------------------------
      [INFO] Reactor Summary:
      [INFO] 
      [INFO] Livy Project Parent POM 0.6.0-incubating ........... SUCCESS [ 39.874 s]
      [INFO] livy-api ........................................... SUCCESS [ 34.074 s]
      [INFO] livy-client-common ................................. SUCCESS [  4.428 s]
      [INFO] livy-test-lib ...................................... SUCCESS [ 11.555 s]
      [INFO] livy-rsc ........................................... SUCCESS [ 17.159 s]
      [INFO] multi-scala-project-root ........................... SUCCESS [  0.437 s]
      [INFO] livy-core-parent ................................... SUCCESS [  1.279 s]
      [INFO] livy-core_2.11 ..................................... SUCCESS [ 19.109 s]
      [INFO] livy-repl-parent ................................... SUCCESS [  7.326 s]
      [INFO] livy-repl_2.11 ..................................... SUCCESS [ 52.145 s]
      [INFO] livy-server ........................................ SUCCESS [01:28 min]
      [INFO] livy-assembly ...................................... SUCCESS [  8.376 s]
      [INFO] livy-client-http ................................... SUCCESS [ 22.525 s]
      [INFO] livy-scala-api-parent .............................. SUCCESS [ 27.709 s]
      [INFO] livy-scala-api_2.11 ................................ SUCCESS [ 25.593 s]
      [INFO] livy-integration-test .............................. SUCCESS [01:15 min]
      [INFO] livy-coverage-report ............................... SUCCESS [  5.768 s]
      [INFO] livy-examples ...................................... SUCCESS [ 25.137 s]
      [INFO] livy-python-api 0.6.0-incubating ................... FAILURE [  1.678 s]
      [INFO] ------------------------------------------------------------------------
      [INFO] BUILD FAILURE
      [INFO] ------------------------------------------------------------------------
      [INFO] Total time: 07:48 min
      [INFO] Finished at: 2020-04-03T14:56:17Z
      [INFO] ------------------------------------------------------------------------
      [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2.1:exec (python-api install) on project livy-python-api: Command execution failed.: Process exited with an error: 1 (Exit value: 1) -> [Help 1]
      [ERROR] 
      [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
      [ERROR] Re-run Maven using the -X switch to enable full debug logging.
      [ERROR] 
      [ERROR] For more information about the errors and possible solutions, please read the following articles:
      [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
      [ERROR] 
      [ERROR] After correcting the problems, you can resume the build with the command
      [ERROR]   mvn <goals> -rf :livy-python-api
      error: Bad exit status from /var/tmp/rpm-tmp.YCK4qE (%build)
          Bad exit status from /var/tmp/rpm-tmp.YCK4qE (%build)
      
      
      RPM build errors:
      
      > Task :livy-rpm FAILED
      
      FAILURE: Build failed with an exception.
      
      * Where:
      Script '/home/vagrant/bigtop/packages.gradle' line: 500
      
      * What went wrong:
      Execution failed for task ':livy-rpm'.
      > Process 'command 'rpmbuild'' finished with non-zero exit value 1
      
      * Try:
      Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
      
      * Get more help at https://help.gradle.org
      
      BUILD FAILED in 8m 17s
      21 actionable tasks: 21 executed
      

      In addition, CentOS 8 and Fedora 31 don't have the python command (they have the python2 command instead), so the build fails as follows:

      centos-8
      [INFO] ------------------< org.apache.livy:livy-python-api >-------------------
      [INFO] Building livy-python-api 0.6.0-incubating                        [19/19]
      [INFO] --------------------------------[ pom ]---------------------------------
      Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/mojo/exec-maven-plugin/1.2.1/exec-maven-plugin-1.2.1.pom
      Progress (1): 7.7 kB    UTING [9m 19s]
      Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/mojo/exec-maven-plugin/1.2.1/exec-maven-plugin-1.2.1.pom (7.7 kB at 34 kB/s)
      Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/mojo/exec-maven-plugin/1.2.1/exec-maven-plugin-1.2.1.jar
      Progress (1): 16/38 kB CUTING [9m 19s]
      Progress (1): 33/38 kB
      Progress (1): 38 kB   
      Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/mojo/exec-maven-plugin/1.2.1/exec-maven-plugin-1.2.1.jar (38 kB at 163 kB/s)
      [INFO] 
      [INFO] --- jacoco-maven-plugin:0.8.2:prepare-agent (default) @ livy-python-api ---
      [INFO] Skipping JaCoCo execution because property jacoco.skip is set.
      [INFO] argLine set to empty
      [INFO] 
      [INFO] --- maven-remote-resources-plugin:1.5:process (process-resource-bundles) @ livy-python-api ---
      [INFO] 
      [INFO] --- scala-maven-plugin:3.2.2:compile (scala-compile-first) @ livy-python-api ---
      [INFO] No sources to compile
      [INFO] 
      [INFO] --- exec-maven-plugin:1.2.1:exec (python-api install) @ livy-python-api ---
      [INFO] ------------------------------------------------------------------------
      [INFO] Reactor Summary:
      [INFO] 
      [INFO] Livy Project Parent POM 0.6.0-incubating ........... SUCCESS [ 51.622 s]
      [INFO] livy-api ........................................... SUCCESS [ 32.867 s]
      [INFO] livy-client-common ................................. SUCCESS [  5.211 s]
      [INFO] livy-test-lib ...................................... SUCCESS [ 12.630 s]
      [INFO] livy-rsc ........................................... SUCCESS [ 16.866 s]
      [INFO] multi-scala-project-root ........................... SUCCESS [  0.628 s]
      [INFO] livy-core-parent ................................... SUCCESS [  1.344 s]
      [INFO] livy-core_2.11 ..................................... SUCCESS [ 22.225 s]
      [INFO] livy-repl-parent ................................... SUCCESS [  7.747 s]
      [INFO] livy-repl_2.11 ..................................... SUCCESS [01:01 min]
      [INFO] livy-server ........................................ SUCCESS [01:38 min]
      [INFO] livy-assembly ...................................... SUCCESS [  8.858 s]
      [INFO] livy-client-http ................................... SUCCESS [ 25.649 s]
      [INFO] livy-scala-api-parent .............................. SUCCESS [ 29.462 s]
      [INFO] livy-scala-api_2.11 ................................ SUCCESS [ 29.101 s]
      [INFO] livy-integration-test .............................. SUCCESS [01:37 min]
      [INFO] livy-coverage-report ............................... SUCCESS [  5.936 s]
      [INFO] livy-examples ...................................... SUCCESS [ 26.238 s]
      [INFO] livy-python-api 0.6.0-incubating ................... FAILURE [  0.673 s]
      [INFO] ------------------------------------------------------------------------
      [INFO] BUILD FAILURE
      [INFO] ------------------------------------------------------------------------
      [INFO] Total time: 08:55 min
      [INFO] Finished at: 2020-04-03T15:19:40Z
      [INFO] ------------------------------------------------------------------------
      [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2.1:exec (python-api install) on project livy-python-api: Command execution failed.: Cannot run program "python" (in directory "/home/vagrant/bigtop/build/livy/rpm/BUILD/livy-0.6.0/python-api"): error=2, No such file or directory -> [Help 1]
      [ERROR] 
      [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
      [ERROR] Re-run Maven using the -X switch to enable full debug logging.
      [ERROR] 
      [ERROR] For more information about the errors and possible solutions, please read the following articles:
      [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
      [ERROR] 
      [ERROR] After correcting the problems, you can resume the build with the command
      [ERROR]   mvn <goals> -rf :livy-python-api
      error: Bad exit status from /var/tmp/rpm-tmp.CAmGp1 (%build)
          Bad exit status from /var/tmp/rpm-tmp.CAmGp1 (%build)
      
      
      RPM build errors:
      
      > Task :livy-rpm FAILED
      
      FAILURE: Build failed with an exception.
      
      * Where:
      Script '/home/vagrant/bigtop/packages.gradle' line: 500
      
      * What went wrong:
      Execution failed for task ':livy-rpm'.
      > Process 'command 'rpmbuild'' finished with non-zero exit value 1
      
      * Try:
      Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
      
      * Get more help at https://help.gradle.org
      
      BUILD FAILED in 9m 24s
      21 actionable tasks: 21 executed
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                sekikn Kengo Seki
                Reporter:
                sekikn Kengo Seki
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 40m
                  40m