Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-4570

Impala shell tarball is not usable on systems with setuptools versions where '0.7' is a substring of the full version string

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: Impala 2.3.0, Impala 2.5.0, Impala 2.4.0, Impala 2.6.0, Impala 2.7.0, Impala 2.8.0
    • Fix Version/s: Impala 2.8.0
    • Component/s: Clients
    • Labels:
      None

      Description

      We saw this error:

      ValueError: A 0.7-series setuptools cannot be installed with distribute. Found one at /usr/lib/python2.7/dist-packages
      

      On a system with setuptools version 20.7.0:

      $ find / -name 'setuptools*' 2>/dev/null
      /usr/lib/python2.7/dist-packages/setuptools
      /usr/lib/python2.7/dist-packages/setuptools-20.7.0.egg-info
      /usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/pip/utils/setuptools_build.py
      /usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/setuptools
      /usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/setuptools-29.0.1.dist-info
      /usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/setuptools.pth
      /usr/lib/cmf/agent/build/env/share/python-wheels/setuptools-20.7.0-py2.py3-none-any.whl
      /usr/lib/hue/tools/virtual-bootstrap/virtualenv_support/setuptools-12.0.5-py2.py3-none-any.whl
      /usr/lib/hue/build/env/lib/python2.7/site-packages/setuptools
      /usr/lib/hue/build/env/lib/python2.7/site-packages/setuptools-12.0.5.dist-info
      /usr/share/doc/python-setuptools/setuptools.txt.gz
      

      It looks like this is a bug in pkg_resources.py that others have encountered, which we obtained from setuptools.

        Activity

        Show
        tarmstrong Tim Armstrong added a comment - Context: https://github.com/pypa/setuptools/issues/543
        Hide
        tarmstrong Tim Armstrong added a comment -

        Forgot to close this

        commit cdbcdca67018fe77ee7a33a11695ca89cf8d476a
        Author: Tim Armstrong <tarmstrong@cloudera.com>
        Date: Thu Dec 1 16:45:15 2016 -0800

        IMPALA-4570: shell tarball breaks with certain setuptools versions

        The bug was in the third-party pkg_resources.py script. The version
        check was broken because it matches any version with a "0.7" substring
        instead of just versions starting with 0.7.

        This is a known bug. setuptools even re-released 20.7.0 as version
        20.8.0 to avoid it:
        https://github.com/pypa/setuptools/commit/e5822f0d5be6386bf86cde03988bfdf1bfc2e935

        Testing:
        I was unable to reproduce this locally, but I think the fix is clear-cut
        enough that this is ok.

        Change-Id: I0565c0e6c1be7d82c3f35d2545ba044a684bb075
        Reviewed-on: http://gerrit.cloudera.org:8080/5314
        Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
        Tested-by: Impala Public Jenkins

        Show
        tarmstrong Tim Armstrong added a comment - Forgot to close this commit cdbcdca67018fe77ee7a33a11695ca89cf8d476a Author: Tim Armstrong <tarmstrong@cloudera.com> Date: Thu Dec 1 16:45:15 2016 -0800 IMPALA-4570 : shell tarball breaks with certain setuptools versions The bug was in the third-party pkg_resources.py script. The version check was broken because it matches any version with a "0.7" substring instead of just versions starting with 0.7. This is a known bug. setuptools even re-released 20.7.0 as version 20.8.0 to avoid it: https://github.com/pypa/setuptools/commit/e5822f0d5be6386bf86cde03988bfdf1bfc2e935 Testing: I was unable to reproduce this locally, but I think the fix is clear-cut enough that this is ok. Change-Id: I0565c0e6c1be7d82c3f35d2545ba044a684bb075 Reviewed-on: http://gerrit.cloudera.org:8080/5314 Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com> Tested-by: Impala Public Jenkins
        Hide
        jbapple Jim Apple added a comment -

        This is a bulk comment on all issues with Fix Version 2.8.0 that were resolved on or after 2016-12-09.

        2.8.0 was branched on December 9, with only two changes to master cherry-picked to the 2.8.0 release branch after that:

        https://github.com/apache/incubator-impala/commits/2.8.0

        Issues fixed after December 9 might not be fixed in 2.8.0. If you are the one who marked this issue Resolved, can you check to see if the patch is in 2.8.0 by using the link above? If the patch is not in 2.8.0, can you change the Fix Version to 2.9.0?

        Thank you!

        Show
        jbapple Jim Apple added a comment - This is a bulk comment on all issues with Fix Version 2.8.0 that were resolved on or after 2016-12-09. 2.8.0 was branched on December 9, with only two changes to master cherry-picked to the 2.8.0 release branch after that: https://github.com/apache/incubator-impala/commits/2.8.0 Issues fixed after December 9 might not be fixed in 2.8.0. If you are the one who marked this issue Resolved, can you check to see if the patch is in 2.8.0 by using the link above? If the patch is not in 2.8.0, can you change the Fix Version to 2.9.0? Thank you!
        Hide
        jrussell John Russell added a comment -

        Am I right to think that this is a bug that would only be encountered when building impala-shell, and therefore applies to upstream Apache Impala but not a packaged install through CDH?

        Show
        jrussell John Russell added a comment - Am I right to think that this is a bug that would only be encountered when building impala-shell, and therefore applies to upstream Apache Impala but not a packaged install through CDH?
        Hide
        tarmstrong Tim Armstrong added a comment -

        It applies when installing or using impala-shell. So users with the relevant setuptools version on their local system could hit it.

        Show
        tarmstrong Tim Armstrong added a comment - It applies when installing or using impala-shell. So users with the relevant setuptools version on their local system could hit it.
        Hide
        jrussell John Russell added a comment -

        Added new "known issue" item in this gerrit review: https://gerrit.cloudera.org/#/c/5809/

        Show
        jrussell John Russell added a comment - Added new "known issue" item in this gerrit review: https://gerrit.cloudera.org/#/c/5809/

          People

          • Assignee:
            tarmstrong Tim Armstrong
            Reporter:
            tarmstrong Tim Armstrong
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development