Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-3123 Define Bigtop 1.5.0 release BOM
  3. BIGTOP-3375

Bump Livy to 0.7.0 and build it with Python3

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.5.0
    • Component/s: None
    • Labels:
      None

      Description

      We changed the way to install flake8 and setuptools to use Python3 instead of Python2 in BIGTOP-3364, but I just noticed Livy 0.6.0 only supports Python 2.x, so running ./gradlew livy-pkg-ind fails as follows.

      $ ./gradlew livy-clean livy-pkg-ind
      
      (snip)
      
      Installed /bigtop/build/livy/rpm/BUILD/livy-0.6.0/python-api/UNKNOWN-0.0.0-py2.7.egg
      warning: no previously-included files matching '*.pyc' found anywhere in distribution
      no previously-included directories found matching 'docs/build/'
      warning: install_lib: 'build/lib' does not exist -- no Python modules to install
      
      zip_safe flag not set; analyzing archive contents...
      Traceback (most recent call last):
        File "setup.py", line 57, in <module>
          tests_require=['pytest']
        File "/usr/lib64/python2.7/distutils/core.py", line 112, in setup
          _setup_distribution = dist = klass(attrs)
        File "/usr/lib/python2.7/site-packages/setuptools/dist.py", line 265, in __init__
          self.fetch_build_eggs(attrs.pop('setup_requires'))
        File "/usr/lib/python2.7/site-packages/setuptools/dist.py", line 289, in fetch_build_eggs
          parse_requirements(requires), installer=self.fetch_build_egg
        File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 626, in resolve
          raise DistributionNotFound(req)
      pkg_resources.DistributionNotFound: flake8
      
      (snip)
      
      [INFO] livy-examples ...................................... SUCCESS [  9.374 s]
      [INFO] livy-python-api 0.6.0-incubating ................... FAILURE [  2.545 s]
      [INFO] ------------------------------------------------------------------------
      [INFO] BUILD FAILURE
      [INFO] ------------------------------------------------------------------------
      [INFO] Total time: 10:10 min
      [INFO] Finished at: 2020-07-08T23:20:10Z
      [INFO] ------------------------------------------------------------------------
      [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2.1:exec (python-api install) on project livy-python-api: Command execution failed.: Process exited with an error: 1 (Exit value: 1) -> [Help 1]
      

      I tried to install the python-flake8 package to avoid this, but I got another error:

      [INFO] --- exec-maven-plugin:1.2.1:exec (python-api install) @ livy-python-api ---
      Traceback (most recent call last):
        File "setup.py", line 57, in <module>
          tests_require=['pytest']
        File "/usr/lib64/python2.7/distutils/core.py", line 112, in setup
          _setup_distribution = dist = klass(attrs)
        File "/usr/lib/python2.7/site-packages/setuptools/dist.py", line 265, in __init__
          self.fetch_build_eggs(attrs.pop('setup_requires'))
        File "/usr/lib/python2.7/site-packages/setuptools/dist.py", line 289, in fetch_build_eggs
          parse_requirements(requires), installer=self.fetch_build_egg
        File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 618, in resolve
          dist = best[req.key] = env.best_match(req, self, installer)
        File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 862, in best_match
          return self.obtain(req, installer) # try and download/install
        File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 874, in obtain
          return installer(requirement)
        File "/usr/lib/python2.7/site-packages/setuptools/dist.py", line 339, in fetch_build_egg
          return cmd.easy_install(req)
        File "/usr/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 623, in easy_install
          return self.install_item(spec, dist.location, tmpdir, deps)
        File "/usr/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 653, in install_item
          dists = self.install_eggs(spec, download, tmpdir)
        File "/usr/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 849, in install_eggs
          return self.build_and_install(setup_script, setup_base)
        File "/usr/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1130, in build_and_install
          self.run_setup(setup_script, setup_base, args)
        File "/usr/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1115, in run_setup
          run_setup(setup_script, args)
        File "/usr/lib/python2.7/site-packages/setuptools/sandbox.py", line 69, in run_setup
          lambda: execfile(
        File "/usr/lib/python2.7/site-packages/setuptools/sandbox.py", line 120, in run
          return func()
        File "/usr/lib/python2.7/site-packages/setuptools/sandbox.py", line 71, in <lambda>
          {'__file__':setup_script, '__name__':'__main__'}
        File "setup.py", line 21, in <module>
          
        File "/usr/lib64/python2.7/distutils/core.py", line 112, in setup
          _setup_distribution = dist = klass(attrs)
        File "/usr/lib/python2.7/site-packages/setuptools/dist.py", line 269, in __init__
          _Distribution.__init__(self,attrs)
        File "/usr/lib64/python2.7/distutils/dist.py", line 287, in __init__
          self.finalize_options()
        File "/usr/lib/python2.7/site-packages/setuptools/dist.py", line 302, in finalize_options
          ep.load()(self, ep.name, value)
        File "build/bdist.linux-x86_64/egg/setuptools_scm/integration.py", line 9, in version_keyword
        File "build/bdist.linux-x86_64/egg/setuptools_scm/version.py", line 60, in _warn_if_setuptools_outdated
      setuptools_scm.version.SetuptoolsOutdatedWarning: your setuptools is too old (<12)
      
      (snip)
      
      [INFO] livy-examples ...................................... SUCCESS [  4.536 s]
      [INFO] livy-python-api 0.6.0-incubating ................... FAILURE [  0.887 s]
      [INFO] ------------------------------------------------------------------------
      [INFO] BUILD FAILURE
      [INFO] ------------------------------------------------------------------------
      [INFO] Total time: 01:21 min
      [INFO] Finished at: 2020-07-09T22:57:06Z
      [INFO] ------------------------------------------------------------------------
      [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2.1:exec (python-api install) on project livy-python-api: Command execution failed.: Process exited with an error: 1 (Exit value: 1) -> [Help 1]
      

      CentOS 7 provides the python-setuptools package, but its version is 0.9.8. So we may have to install setuptools as a Python2 library via pip2, as we did before.
      As Yuqi mentioned before, the latest version of setuptools has already dropped Python2 support, but prior to v45.0.0 seems to work with Python2.

      https://setuptools.readthedocs.io/en/latest/history.html#v45-0-0

      So I'll try to revert BIGTOP-3364 and pin setuptools' version to 44.0.0, and check if both Zeppelin and Livy are successfully built on all distros.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                sekikn Kengo Seki
                Reporter:
                sekikn Kengo Seki
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 40m
                  40m