[IMPALA-8754] S3 with S3Guard tests encounter "ResourceNotFoundException" from DynamoDB

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Duplicate
• Affects Version/s: Impala 3.3.0
• Component/s: Infrastructure
• Labels: broken-build flaky
• Epic Color: ghx-label-8
• Assignee: Joe McDonnell
• Reporter: Joe McDonnell
• Created: 12/Jul/19 00:14
• Updated: 23/Jul/20 16:04
• Resolved: 23/Jul/20 16:04

Description

When running tests on s3 with s3guard, various tests can encounter the following error coming from the DynamoDB:

E    Query aborted:Disk I/O error on impala-ec2-centos74-m5-4xlarge-ondemand-02c8.vpc.cloudera.com:22002: Failed to open HDFS file s3a://impala-test-uswest2-1/test-warehouse/tpcds.store_sales_parquet/ss_sold_date_sk=2451718/6843d8a91fc5ae1d-88b2af4b00000004_156969840_data.0.parq
E   Error(2): No such file or directory
E   Root cause: ResourceNotFoundException: Requested resource not found (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ResourceNotFoundException; Request ID: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX)

Tests that have seen this (this is flaky):

• TestTpcdsQuery.test_tpcds_count
• TestHdfsFdCaching.test_caching_disabled_by_param
• TestMtDop.test_compute_stats
• TestScanRangeLengths.test_scan_ranges

Issue Links

• duplicates: IMPALA-9058 S3 tests failing with FileNotFoundException getVersionMarkerItem on ../VERSION
• is duplicated by: IMPALA-9996 An S3 test failing with ResourceNotFoundException
• is related to: IMPALA-9461 S3Guard metadata store cannot be found