[IMPALA-6517] bootstrap_toolchain.py fails to recognize lsb_release output from RHEL

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: Impala 2.12.0
• Epic Color: ghx-label-1
• Assignee: Vincent Tran
• Reporter: Vincent Tran
• Created: 13/Feb/18 20:48
• Updated: 22/Feb/18 21:27
• Resolved: 22/Feb/18 21:27

Description

 

[root@legion-3 Impala]# lsb_release -irs
RedHatEnterpriseServer 6.9
[root@legion-3 Impala]# bin/impala-python bin/bootstrap_toolchain.py
2018-02-13 15:10:55,029 MainThread ERROR: The following packages are not in their expected locations.
 <__main__.Package object at 0x7fc08aad68d0> (expected directory /root/Impala/toolchain/llvm-3.9.1-asserts to exist)
 <__main__.Package object at 0x7fc08aad6910> (expected directory /root/Impala/toolchain/llvm-3.9.1 to exist)
 <__main__.Package object at 0x7fc08aad6950> (expected directory /root/Impala/toolchain/kudu-1520b39 to exist)
 <__main__.Package object at 0x7fc08aad6990> (expected directory /root/Impala/toolchain/avro-1.7.4-p4 to exist)
 <__main__.Package object at 0x7fc08aad69d0> (expected directory /root/Impala/toolchain/binutils-2.26.1 to exist)
 <__main__.Package object at 0x7fc08aad6a50> (expected directory /root/Impala/toolchain/boost-1.57.0-p3 to exist)
 <__main__.Package object at 0x7fc08aad6a10> (expected directory /root/Impala/toolchain/breakpad-97a98836768f8f0154f8f86e5e14c2bb7e74132e-p2 to exist)
 <__main__.Package object at 0x7fc08aad6ad0> (expected directory /root/Impala/toolchain/bzip2-1.0.6-p2 to exist)
 <__main__.Package object at 0x7fc08aad6a90> (expected directory /root/Impala/toolchain/cmake-3.8.2-p1 to exist)
 <__main__.Package object at 0x7fc08aad6b10> (expected directory /root/Impala/toolchain/crcutil-440ba7babeff77ffad992df3a10c767f184e946e-p1 to exist)
 <__main__.Package object at 0x7fc08aad6b90> (expected directory /root/Impala/toolchain/flatbuffers-1.6.0 to exist)
 <__main__.Package object at 0x7fc08aad6b50> (expected directory /root/Impala/toolchain/gcc-4.9.2 to exist)
 <__main__.Package object at 0x7fc08aad6bd0> (expected directory /root/Impala/toolchain/gflags-2.2.0-p1 to exist)
 <__main__.Package object at 0x7fc08aad6c50> (expected directory /root/Impala/toolchain/glog-0.3.4-p2 to exist)
 <__main__.Package object at 0x7fc08aad6c10> (expected directory /root/Impala/toolchain/gperftools-2.5 to exist)
 <__main__.Package object at 0x7fc08aad6c90> (expected directory /root/Impala/toolchain/gtest-1.6.0 to exist)
 <__main__.Package object at 0x7fc08aad6cd0> (expected directory /root/Impala/toolchain/libev-4.20 to exist)
 <__main__.Package object at 0x7fc08aad6d10> (expected directory /root/Impala/toolchain/lz4-1.7.5 to exist)
 <__main__.Package object at 0x7fc08aad6d50> (expected directory /root/Impala/toolchain/openldap-2.4.25 to exist)
 <__main__.Package object at 0x7fc08aad6dd0> (expected directory /root/Impala/toolchain/openssl-1.0.2l to exist)
 <__main__.Package object at 0x7fc08aad6e10> (expected directory /root/Impala/toolchain/protobuf-2.6.1 to exist)
 <__main__.Package object at 0x7fc08aad6e50> (expected directory /root/Impala/toolchain/rapidjson-0.11 to exist)
 <__main__.Package object at 0x7fc08aad6e90> (expected directory /root/Impala/toolchain/re2-20130115-p1 to exist)
 <__main__.Package object at 0x7fc08aad6d90> (expected directory /root/Impala/toolchain/snappy-1.1.4 to exist)
 <__main__.Package object at 0x7fc08aad6f10> (expected directory /root/Impala/toolchain/thrift-0.9.0-p11 to exist)
 <__main__.Package object at 0x7fc08aad6f50> (expected directory /root/Impala/toolchain/tpc-h-2.17.0 to exist)
 <__main__.Package object at 0x7fc08aad6ed0> (expected directory /root/Impala/toolchain/tpc-ds-2.1.0 to exist)
 <__main__.Package object at 0x7fc08aad6fd0> (expected directory /root/Impala/toolchain/zlib-1.2.8 to exist)
Pre-built toolchain archives not available for your platform.
Clone and build native toolchain from source using this repository:
 https://github.com/cloudera/native-toolchain
Traceback (most recent call last):
 File "bin/bootstrap_toolchain.py", line 427, in <module>
 bootstrap(toolchain_root, packages)
 File "bin/bootstrap_toolchain.py", line 161, in bootstrap
 check_custom_toolchain(toolchain_root, packages)
 File "bin/bootstrap_toolchain.py", line 213, in check_custom_toolchain
 raise Exception("Toolchain bootstrap failed: required packages were missing")
Exception: Toolchain bootstrap failed: required packages were missing

 
It turns out we currently match the `lsb_release -irs` output against this map:

OS_MAPPING = {
  "centos6" : "ec2-package-centos-6",
  "centos5" : "ec2-package-centos-5",
  "centos7" : "ec2-package-centos-7",
  "debian6" : "ec2-package-debian-6",
  "debian7" : "ec2-package-debian-7",
  "debian8" : "ec2-package-debian-8",
  "suselinux11": "ec2-package-sles-11",
  "suselinux12": "ec2-package-sles-12",
  "suse12.2": "ec2-package-sles-12",
  "ubuntu12.04" : "ec2-package-ubuntu-12-04",
  "ubuntu14.04" : "ec2-package-ubuntu-14-04",
  "ubuntu15.04" : "ec2-package-ubuntu-14-04",
  "ubuntu15.10" : "ec2-package-ubuntu-14-04",
  "ubuntu16.04" : "ec2-package-ubuntu-16-04",
}

We should add redhatenterpriseserver<majorversion> to the map for centos packages and relax the matching rules so that we only need to match against the major version.