[IMPALA-6910] Multiple tests failing on S3 build: error reading from HDFS file

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Cannot Reproduce
• Affects Version/s: Impala 3.0
• Fix Version/s: Impala 3.2.0
• Component/s: Backend
• Labels: broken-build flaky s3
• Epic Color: ghx-label-8
• Assignee: Unassigned
• Reporter: David Knupp
• Created: 23/Apr/18 20:30
• Updated: 12/Feb/20 23:42
• Resolved: 12/Feb/20 23:42

Description

Stacktrace

query_test/test_compressed_formats.py:149: in test_seq_writer
    self.run_test_case('QueryTest/seq-writer', vector, unique_database)
common/impala_test_suite.py:397: in run_test_case
    result = self.__execute_query(target_impalad_client, query, user=user)
common/impala_test_suite.py:612: in __execute_query
    return impalad_client.execute(query, user=user)
common/impala_connection.py:160: in execute
    return self.__beeswax_client.execute(sql_stmt, user=user)
beeswax/impala_beeswax.py:173: in execute
    handle = self.__execute_query(query_string.strip(), user=user)
beeswax/impala_beeswax.py:341: in __execute_query
    self.wait_for_completion(handle)
beeswax/impala_beeswax.py:361: in wait_for_completion
    raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
E   ImpalaBeeswaxException: ImpalaBeeswaxException:
E    Query aborted:Disk I/O error: Error reading from HDFS file: s3a://impala-cdh5-s3-test/test-warehouse/tpcds.store_sales_parquet/ss_sold_date_sk=2452585/a5482dcb946b6c98-7543e0dd00000004_95929617_data.0.parq
E   Error(255): Unknown error 255
E   Root cause: SdkClientException: Data read has a different length than the expected: dataLength=8576; expectedLength=17785; includeSkipped=true; in.getClass()=class com.amazonaws.services.s3.AmazonS3Client$2; markedSupported=false; marked=0; resetSinceLastMarked=false; markCount=0; resetCount=0

Issue Links

• blocks: IMPALA-7070 Failed test: query_test.test_nested_types.TestParquetArrayEncodings.test_thrift_array_of_arrays on S3
• relates to: HADOOP-15541 AWS SDK can mistake stream timeouts for EOF and throw SdkClientExceptions