Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-6517

bootstrap_toolchain.py fails to recognize lsb_release output from RHEL

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: Impala 2.12.0
    • Component/s: None
    • Labels:
      None
    • Epic Color:
      ghx-label-1

      Description

       

      [root@legion-3 Impala]# lsb_release -irs
      RedHatEnterpriseServer 6.9
      [root@legion-3 Impala]# bin/impala-python bin/bootstrap_toolchain.py
      2018-02-13 15:10:55,029 MainThread ERROR: The following packages are not in their expected locations.
       <__main__.Package object at 0x7fc08aad68d0> (expected directory /root/Impala/toolchain/llvm-3.9.1-asserts to exist)
       <__main__.Package object at 0x7fc08aad6910> (expected directory /root/Impala/toolchain/llvm-3.9.1 to exist)
       <__main__.Package object at 0x7fc08aad6950> (expected directory /root/Impala/toolchain/kudu-1520b39 to exist)
       <__main__.Package object at 0x7fc08aad6990> (expected directory /root/Impala/toolchain/avro-1.7.4-p4 to exist)
       <__main__.Package object at 0x7fc08aad69d0> (expected directory /root/Impala/toolchain/binutils-2.26.1 to exist)
       <__main__.Package object at 0x7fc08aad6a50> (expected directory /root/Impala/toolchain/boost-1.57.0-p3 to exist)
       <__main__.Package object at 0x7fc08aad6a10> (expected directory /root/Impala/toolchain/breakpad-97a98836768f8f0154f8f86e5e14c2bb7e74132e-p2 to exist)
       <__main__.Package object at 0x7fc08aad6ad0> (expected directory /root/Impala/toolchain/bzip2-1.0.6-p2 to exist)
       <__main__.Package object at 0x7fc08aad6a90> (expected directory /root/Impala/toolchain/cmake-3.8.2-p1 to exist)
       <__main__.Package object at 0x7fc08aad6b10> (expected directory /root/Impala/toolchain/crcutil-440ba7babeff77ffad992df3a10c767f184e946e-p1 to exist)
       <__main__.Package object at 0x7fc08aad6b90> (expected directory /root/Impala/toolchain/flatbuffers-1.6.0 to exist)
       <__main__.Package object at 0x7fc08aad6b50> (expected directory /root/Impala/toolchain/gcc-4.9.2 to exist)
       <__main__.Package object at 0x7fc08aad6bd0> (expected directory /root/Impala/toolchain/gflags-2.2.0-p1 to exist)
       <__main__.Package object at 0x7fc08aad6c50> (expected directory /root/Impala/toolchain/glog-0.3.4-p2 to exist)
       <__main__.Package object at 0x7fc08aad6c10> (expected directory /root/Impala/toolchain/gperftools-2.5 to exist)
       <__main__.Package object at 0x7fc08aad6c90> (expected directory /root/Impala/toolchain/gtest-1.6.0 to exist)
       <__main__.Package object at 0x7fc08aad6cd0> (expected directory /root/Impala/toolchain/libev-4.20 to exist)
       <__main__.Package object at 0x7fc08aad6d10> (expected directory /root/Impala/toolchain/lz4-1.7.5 to exist)
       <__main__.Package object at 0x7fc08aad6d50> (expected directory /root/Impala/toolchain/openldap-2.4.25 to exist)
       <__main__.Package object at 0x7fc08aad6dd0> (expected directory /root/Impala/toolchain/openssl-1.0.2l to exist)
       <__main__.Package object at 0x7fc08aad6e10> (expected directory /root/Impala/toolchain/protobuf-2.6.1 to exist)
       <__main__.Package object at 0x7fc08aad6e50> (expected directory /root/Impala/toolchain/rapidjson-0.11 to exist)
       <__main__.Package object at 0x7fc08aad6e90> (expected directory /root/Impala/toolchain/re2-20130115-p1 to exist)
       <__main__.Package object at 0x7fc08aad6d90> (expected directory /root/Impala/toolchain/snappy-1.1.4 to exist)
       <__main__.Package object at 0x7fc08aad6f10> (expected directory /root/Impala/toolchain/thrift-0.9.0-p11 to exist)
       <__main__.Package object at 0x7fc08aad6f50> (expected directory /root/Impala/toolchain/tpc-h-2.17.0 to exist)
       <__main__.Package object at 0x7fc08aad6ed0> (expected directory /root/Impala/toolchain/tpc-ds-2.1.0 to exist)
       <__main__.Package object at 0x7fc08aad6fd0> (expected directory /root/Impala/toolchain/zlib-1.2.8 to exist)
      Pre-built toolchain archives not available for your platform.
      Clone and build native toolchain from source using this repository:
       https://github.com/cloudera/native-toolchain
      Traceback (most recent call last):
       File "bin/bootstrap_toolchain.py", line 427, in <module>
       bootstrap(toolchain_root, packages)
       File "bin/bootstrap_toolchain.py", line 161, in bootstrap
       check_custom_toolchain(toolchain_root, packages)
       File "bin/bootstrap_toolchain.py", line 213, in check_custom_toolchain
       raise Exception("Toolchain bootstrap failed: required packages were missing")
      Exception: Toolchain bootstrap failed: required packages were missing
      

       
      It turns out we currently match the `lsb_release -irs` output against this map:

      OS_MAPPING = {
        "centos6" : "ec2-package-centos-6",
        "centos5" : "ec2-package-centos-5",
        "centos7" : "ec2-package-centos-7",
        "debian6" : "ec2-package-debian-6",
        "debian7" : "ec2-package-debian-7",
        "debian8" : "ec2-package-debian-8",
        "suselinux11": "ec2-package-sles-11",
        "suselinux12": "ec2-package-sles-12",
        "suse12.2": "ec2-package-sles-12",
        "ubuntu12.04" : "ec2-package-ubuntu-12-04",
        "ubuntu14.04" : "ec2-package-ubuntu-14-04",
        "ubuntu15.04" : "ec2-package-ubuntu-14-04",
        "ubuntu15.10" : "ec2-package-ubuntu-14-04",
        "ubuntu16.04" : "ec2-package-ubuntu-16-04",
      }
      

      We should add redhatenterpriseserver<majorversion> to the map for centos packages and relax the matching rules so that we only need to match against the major version.

        Attachments

          Activity

            People

            • Assignee:
              thundergun Vincent Tran
              Reporter:
              thundergun Vincent Tran
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: