[BIGTOP-3375] Bump Livy to 0.7.0 and build it with Python3 - ASF JIRA

Details

Type: Sub-task
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: None
Fix Version/s: 1.5.0
Component/s: None
Labels:
None

Description

We changed the way to install flake8 and setuptools to use Python3 instead of Python2 in ~~BIGTOP-3364~~, but I just noticed Livy 0.6.0 only supports Python 2.x, so running ./gradlew livy-pkg-ind fails as follows.

$ ./gradlew livy-clean livy-pkg-ind

(snip)

Installed /bigtop/build/livy/rpm/BUILD/livy-0.6.0/python-api/UNKNOWN-0.0.0-py2.7.egg
warning: no previously-included files matching '*.pyc' found anywhere in distribution
no previously-included directories found matching 'docs/build/'
warning: install_lib: 'build/lib' does not exist -- no Python modules to install

zip_safe flag not set; analyzing archive contents...
Traceback (most recent call last):
  File "setup.py", line 57, in <module>
    tests_require=['pytest']
  File "/usr/lib64/python2.7/distutils/core.py", line 112, in setup
    _setup_distribution = dist = klass(attrs)
  File "/usr/lib/python2.7/site-packages/setuptools/dist.py", line 265, in __init__
    self.fetch_build_eggs(attrs.pop('setup_requires'))
  File "/usr/lib/python2.7/site-packages/setuptools/dist.py", line 289, in fetch_build_eggs
    parse_requirements(requires), installer=self.fetch_build_egg
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 626, in resolve
    raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: flake8

(snip)

[INFO] livy-examples ...................................... SUCCESS [  9.374 s]
[INFO] livy-python-api 0.6.0-incubating ................... FAILURE [  2.545 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 10:10 min
[INFO] Finished at: 2020-07-08T23:20:10Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2.1:exec (python-api install) on project livy-python-api: Command execution failed.: Process exited with an error: 1 (Exit value: 1) -> [Help 1]

I tried to install the python-flake8 package to avoid this, but I got another error:

[INFO] --- exec-maven-plugin:1.2.1:exec (python-api install) @ livy-python-api ---
Traceback (most recent call last):
  File "setup.py", line 57, in <module>
    tests_require=['pytest']
  File "/usr/lib64/python2.7/distutils/core.py", line 112, in setup
    _setup_distribution = dist = klass(attrs)
  File "/usr/lib/python2.7/site-packages/setuptools/dist.py", line 265, in __init__
    self.fetch_build_eggs(attrs.pop('setup_requires'))
  File "/usr/lib/python2.7/site-packages/setuptools/dist.py", line 289, in fetch_build_eggs
    parse_requirements(requires), installer=self.fetch_build_egg
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 618, in resolve
    dist = best[req.key] = env.best_match(req, self, installer)
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 862, in best_match
    return self.obtain(req, installer) # try and download/install
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 874, in obtain
    return installer(requirement)
  File "/usr/lib/python2.7/site-packages/setuptools/dist.py", line 339, in fetch_build_egg
    return cmd.easy_install(req)
  File "/usr/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 623, in easy_install
    return self.install_item(spec, dist.location, tmpdir, deps)
  File "/usr/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 653, in install_item
    dists = self.install_eggs(spec, download, tmpdir)
  File "/usr/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 849, in install_eggs
    return self.build_and_install(setup_script, setup_base)
  File "/usr/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1130, in build_and_install
    self.run_setup(setup_script, setup_base, args)
  File "/usr/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1115, in run_setup
    run_setup(setup_script, args)
  File "/usr/lib/python2.7/site-packages/setuptools/sandbox.py", line 69, in run_setup
    lambda: execfile(
  File "/usr/lib/python2.7/site-packages/setuptools/sandbox.py", line 120, in run
    return func()
  File "/usr/lib/python2.7/site-packages/setuptools/sandbox.py", line 71, in <lambda>
    {'__file__':setup_script, '__name__':'__main__'}
  File "setup.py", line 21, in <module>
    
  File "/usr/lib64/python2.7/distutils/core.py", line 112, in setup
    _setup_distribution = dist = klass(attrs)
  File "/usr/lib/python2.7/site-packages/setuptools/dist.py", line 269, in __init__
    _Distribution.__init__(self,attrs)
  File "/usr/lib64/python2.7/distutils/dist.py", line 287, in __init__
    self.finalize_options()
  File "/usr/lib/python2.7/site-packages/setuptools/dist.py", line 302, in finalize_options
    ep.load()(self, ep.name, value)
  File "build/bdist.linux-x86_64/egg/setuptools_scm/integration.py", line 9, in version_keyword
  File "build/bdist.linux-x86_64/egg/setuptools_scm/version.py", line 60, in _warn_if_setuptools_outdated
setuptools_scm.version.SetuptoolsOutdatedWarning: your setuptools is too old (<12)

(snip)

[INFO] livy-examples ...................................... SUCCESS [  4.536 s]
[INFO] livy-python-api 0.6.0-incubating ................... FAILURE [  0.887 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:21 min
[INFO] Finished at: 2020-07-09T22:57:06Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2.1:exec (python-api install) on project livy-python-api: Command execution failed.: Process exited with an error: 1 (Exit value: 1) -> [Help 1]

CentOS 7 provides the python-setuptools package, but its version is 0.9.8. So we may have to install setuptools as a Python2 library via pip2, as we did before.
As Yuqi mentioned before, the latest version of setuptools has already dropped Python2 support, but prior to v45.0.0 seems to work with Python2.

https://setuptools.readthedocs.io/en/latest/history.html#v45-0-0

So I'll try to revert ~~BIGTOP-3364~~ and pin setuptools' version to 44.0.0, and check if both Zeppelin and Livy are successfully built on all distros.

Bump Livy to 0.7.0 and build it with Python3

Details

Description

Attachments

Issue Links

Activity

People

Dates

Time Tracking