[IMPALA-13072] Toolchain: Add retries for uploading artifacts to the s3 buckets

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: Impala 4.5.0
• Component/s: Infrastructure
• Epic Color: ghx-label-3
• Assignee: Joe McDonnell
• Reporter: Joe McDonnell
• Created: 12/May/24 00:10
• Updated: 14/May/24 16:57

Description

On ARM toolchain builds, we have seen some failures to upload tarballs to s3:

22:17:06 impala-toolchain-redhat8: Uploading /mnt/build/llvm-5.0.1-asserts-p7-gcc-10.4.0.tar.gz to s3://native-toolchain/build/33-f93e2c9a86/llvm/5.0.1-asserts-p7-gcc-10.4.0/llvm-5.0.1-asserts-p7-gcc-10.4.0-ec2-package-centos-8-aarch64.tar.gz
22:17:06 impala-toolchain-redhat8: /mnt/functions.sh: line 385: 680012 Segmentation fault      (core dumped) aws s3 cp --only-show-errors "${PACKAGE_FINAL_TGZ}" "${PACKAGE_S3_DESTINATION}"

Since we do many uploads, even a relatively low failure rate can make it hard to get a passing build. We should change the code to retry the upload.

Issue Links

• relates to: IMPALA-13025 Use AWSCLI v2 for bucket uploads during toolchain builds