[IMPALA-10972] Adapt Impala's tests to behavior change in HIVE-24920

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: Impala 4.1.0
• Fix Version/s: Impala 4.1.0
• Component/s: Infrastructure
• Labels: broken-build
• Target Version: Impala 4.1.0
• Epic Color: ghx-label-9
• Assignee: Joe McDonnell
• Reporter: Joe McDonnell
• Created: 18/Oct/21 19:25
• Updated: 17/Nov/21 20:57
• Resolved: 17/Nov/21 20:57

Description

In HIVE-24920, the behavior changed so that creating a managed table that points to an existing location without specifying the location explicitly will give an error.

This impacts a lot of our tests that use tests/common/file_utils.py's create_table_from_file() or create_table_and_copy_files(), causing a lot of test issues like this:

E   ImpalaBeeswaxException: ImpalaBeeswaxException:
E    INNER EXCEPTION: <class 'beeswaxd.ttypes.BeeswaxException'>
E    MESSAGE: ImpalaRuntimeException: Error making 'createTable' RPC to Hive Metastore: 
E   CAUSED BY: MetaException: Exception determining external table location:Default location is not available for table: hdfs://localhost:20500/test-warehouse/test_error_propagation_race_ccd6c80c.db/bad_magic_number

tests/common/file_utils.py populates a table's directory before using a normal create table without the location specified. This is now prohibited.

tests/common/file_utils.py behaves this way as an attempt to make s3 more consistent back when it did not have full consistency. See  IMPALA-7804: https://github.com/apache/impala/commit/f1a3c47959e17d301f970b7ea2755067d00ae986

This is no longer a concern, so these functions should change back to something compatible with HIVE-24920.

Marking this as a blocker, because it prevents moving to a new Hive.