[IMPALA-8344] Add support for running S3 tests with S3Guard

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 3.3.0
• Fix Version/s: Impala 3.3.0
• Component/s: Infrastructure
• Epic Color: ghx-label-2
• Assignee: Joe McDonnell
• Reporter: Joe McDonnell
• Created: 25/Mar/19 17:00
• Updated: 23/Dec/20 22:52
• Resolved: 23/May/19 19:41

Description

Impala s3 tests can encounter failures due to s3's consistency model. S3Guard adds consistency to s3 operations to avoid these types of failures:

https://blog.cloudera.com/blog/2017/08/introducing-s3guard-s3-consistency-for-apache-hadoop/

Adding support to run tests with S3Guard provides a way to limit flakiness, and it provides coverage for users that would want to use Impala on S3 with S3Guard.

Support will involve adding the appropriate configuration to core-site.xml. In order to maintain the S3Guard index appropriately, file modifications should go through HDFS commands rather than boto s3 commands. Finally, to reduce costs, Impala may want to have a script to purge S3Guard's dynamodb. 

Issue Links

• is related to: IMPALA-7404 query_test.test_delimited_text.TestDelimitedText.test_delimited_text_newlines fails to return any rows
• is related to: IMPALA-7733 TestInsertParquetQueries.test_insert_parquet is flaky in S3 due to rename
• is related to: IMPALA-8500 test_timestamp_out_of_range fails with NoSuchObjectException: test_timestamp_out_of_range_dc37915d on S3
• relates to: IMPALA-7153 filesystem_client methods expect unusual paths
• relates to: IMPALA-8950 Add -d and -f option to copyFromLocal and re-enable disabled S3 tests