[IMPALA-10199] Need to update bootstrap_toolchain.py with the 20.04 version of toolchain file when available

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: Impala 4.2.0
• Component/s: Infrastructure
• Epic Color: ghx-label-10
• Assignee: Joe McDonnell
• Reporter: Qifan Chen
• Created: 30/Sep/20 17:56
• Updated: 20/Dec/22 18:38
• Resolved: 25/May/22 17:27

Description

Currently, for Ubuntu 20.04, the impala development environment uses the toolchain of 18.04, since the toolchain for 20.04 does not exist in EC2, in bin/bootstrap_toolchain.py.

 

Maps return values from 'lsb_release -irs' to the corresponding OS labels for both the 
# toolchain and the CDP components. 
OsMapping = namedtuple('OsMapping', ['lsb_release', 'toolchain', 'cdh']) 
OS_MAPPING = [ 
 OsMapping("centos5", "ec2-package-centos-5", None), 
 OsMapping("centos6", "ec2-package-centos-6", "redhat6"), 
 OsMapping("centos7", "ec2-package-centos-7", "redhat7"), 
 OsMapping("centos8", "ec2-package-centos-8", "redhat8"), 
 OsMapping("redhatenterpriseserver5", "ec2-package-centos-5", None), 
 OsMapping("redhatenterpriseserver6", "ec2-package-centos-6", "redhat6"), 
 OsMapping("redhatenterpriseserver7", "ec2-package-centos-7", "redhat7"), 
 OsMapping("redhatenterpriseserver8", "ec2-package-centos-8", "redhat8"), 
 OsMapping("debian6", "ec2-package-debian-6", None), 
 OsMapping("debian7", "ec2-package-debian-7", None), 
 OsMapping("debian8", "ec2-package-debian-8", "debian8"), 
 OsMapping("suselinux11", "ec2-package-sles-11", None), 
 OsMapping("suselinux12", "ec2-package-sles-12", "sles12"), 
 OsMapping("suse12", "ec2-package-sles-12", "sles12"), 
 OsMapping("ubuntu12.04", "ec2-package-ubuntu-12-04", None), 
 OsMapping("ubuntu14.04", "ec2-package-ubuntu-14-04", None), 
 OsMapping("ubuntu15.04", "ec2-package-ubuntu-14-04", None), 
 OsMapping("ubuntu15.10", "ec2-package-ubuntu-14-04", None), 
 OsMapping('ubuntu16.04', "ec2-package-ubuntu-16-04", "ubuntu1604"), 
 OsMapping('ubuntu18.04', "ec2-package-ubuntu-18-04", "ubuntu1804"), 
 OsMapping('ubuntu20.04', "ec2-package-ubuntu-18-04", "ubuntu1804") 
]

 

Error was retured otherwise. 

 

2020-09-28 11:09:16,660 Thread-3 INFO: Downloading https://native-toolchain.s3.amazonaws.com/build/55-dcf54c8601/gcc/7.5.0-gcc-7.5.0/gcc-7.5.0-gcc-7.5.0-ec2-package-ubuntu-20-04.tar.gz to /home/qchen/Impala/toolchain/toolchain-packages-gcc7.5.0/gcc-7.5.0-gcc-7.5.0-ec2-package-ubuntu-20-04.tar.gz (attempt 1)
2020-09-28 11:09:16,667 Thread-4 INFO: Downloading https://native-toolchain.s3.amazonaws.com/build/55-dcf54c8601/llvm/5.0.1-asserts-p3-gcc-7.5.0/llvm-5.0.1-asserts-p3-gcc-7.5.0-ec2-package-ubuntu-20-04.tar.gz to /home/qchen/Impala/toolchain/toolchain-packages-gcc7.5.0/llvm-5.0.1-asserts-p3-gcc-7.5.0-ec2-package-ubuntu-20-04.tar.gz (attempt 1)
2020-09-28 11:09:16,668 Thread-2 INFO: Downloading https://native-toolchain.s3.amazonaws.com/build/55-dcf54c8601/llvm/5.0.1-p3-gcc-7.5.0/llvm-5.0.1-p3-gcc-7.5.0-ec2-package-ubuntu-20-04.tar.gz to /home/qchen/Impala/toolchain/toolchain-packages-gcc7.5.0/llvm-5.0.1-p3-gcc-7.5.0-ec2-package-ubuntu-20-04.tar.gz (attempt 1)
2020-09-28 11:09:17,002 Thread-1 ERROR: Download failed; retrying after sleep: Command with args '['wget', 'https://native-toolchain.s3.amazonaws.com/build/55-dcf54c8601/avro/1.7.4-p5-gcc-7.5.0/avro-1.7.4-p5-gcc-7.5.0-ec2-package-ubuntu-20-04.tar.gz', '--directory-prefix=/home/qchen/Impala/toolchain/toolchain-packages-gcc7.5.0']' failed with exit code 8:
--2020-09-28 11:09:16-- https://native-toolchain.s3.amazonaws.com/build/55-dcf54c8601/avro/1.7.4-p5-gcc-7.5.0/avro-1.7.4-p5-gcc-7.5.0-ec2-package-ubuntu-20-04.tar.gz
Resolving native-toolchain.s3.amazonaws.com (native-toolchain.s3.amazonaws.com)... 52.219.116.82
Connecting to native-toolchain.s3.amazonaws.com (native-toolchain.s3.amazonaws.com)|52.219.116.82|:443... connected.

Issue Links

• blocks: IMPALA-10962 Toolchain Python loses readline functionality on Ubuntu 20.04
• is related to: IMPALA-11077 Add first-tier toolchain support for Ubuntu 20.04