[IMPALA-11514] Workaround s3 timeout waiting for connection from pool (HADOOP-18410)

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: Impala 4.2.0
• Fix Version/s: Impala 4.2.0
• Component/s: Infrastructure
• Labels: broken-build
• Epic Color: ghx-label-5
• Assignee: Joe McDonnell
• Reporter: Joe McDonnell
• Created: 19/Aug/22 23:29
• Updated: 31/Oct/22 04:48
• Resolved: 23/Aug/22 21:12

Description

When testing on s3, we see dataload fail when trying to load testcases:

12:00:17 Creating tpcds testcase data (logging to /data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/logs/data_loading/create-tpcds-testcase-data.log)... 
12:00:17     FAILED (Took: 0 min 13 sec)
12:00:30     '/data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/testdata/bin/create-tpcds-testcase-files.sh' failed. Tail of log:
12:00:30  order by t_s_secyear.customer_id
12:00:30          ,t_s_secyear.customer_first_name
12:00:30          ,t_s_secyear.customer_last_name
12:00:30          ,t_s_secyear.customer_email_address
12:00:30 limit 100
12:00:30 Query submitted at: 2022-08-18 12:00:25 (Coordinator: http://hostname:25000)
12:00:30 ERROR: AnalysisException: getFileStatus on s3a://bucketname/test-warehouse/tpcds-testcase-data: com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
12:00:30 CAUSED BY: InterruptedIOException: getFileStatus on s3a://bucketname/test-warehouse/tpcds-testcase-data: com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
12:00:30 CAUSED BY: SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
12:00:30 CAUSED BY: ConnectionPoolTimeoutException: Timeout waiting for connection from pool

This has been tracked down to https://issues.apache.org/jira/browse/HADOOP-18410

A temporary workaround is to specify fs.s3a.input.async.drain.threshold=512G in core-site.xml.

We should work around this issue until the fix arrives.