[IMPALA-5330] Impala tests never use or set a secondary FS, so TestMultipleFilesystems is always skipped

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Infrastructure
• Labels: s3 test
• Epic Color: ghx-label-3
• Assignee: Unassigned
• Reporter: David Knupp
• Created: 16/May/17 00:18
• Updated: 08/Jul/17 23:49

Description

This issue came up while investigating how can set up Impala + ADLS testing on remote clusters. Since testing Impala with ADLS is analogous to testing Impala with s3, I used that testing as a starting point.

When we change the TARGET_FILESYSTEM to 's3', the end result is to overwrite the defaultFS setting in core-site.xml to the s3 location. There's not a point (or doesn't appear to be a point) where we ever treat s3 as a true secondary filesystem.

Further more, if we look at our test suite that is supposed to check multiple filesystems simultaneously, we can see there's a pytest SkipIf decorator set to skip the test if a secondary FS is not set.

@SkipIf.no_secondary_fs
class TestMultipleFilesystems(ImpalaTestSuite)

From what I can see by going through old Jenkins console logs, this test never gets executed. It's skipped during exhaustive runs with HDFS as the default, and is also skipped during s3 and local FS test runs as well.

04:27:57 query_test/test_multiple_filesystems.py::TestMultipleFilesystems::test_multiple_filesystems[exec_option: {'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0, 'batch_size': 0, 'num_nodes': 0} | table_format: text/none] SKIPPED