[IMPALA-7153] filesystem_client methods expect unusual paths

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 3.0
• Fix Version/s: Impala 3.3.0
• Component/s: Infrastructure
• Labels: ramp-up
• Epic Color: ghx-label-9
• Assignee: Unassigned
• Reporter: Daniel Hecht
• Created: 08/Jun/18 16:04
• Updated: 23/Dec/20 22:52
• Resolved: 23/Dec/20 22:52

Description

In Impala end to end python tests, most paths are either absolute or fully qualified (i.e. /test-warehouse/db/table or hdfs://namenode:port/test-warehouse/db/table), yet the classes derived from BaseFilesystem tend to expect paths that are absolute but without the leading slash.

This leads to awkward and error prone testing code. For example, see IMPALA-7099 and test_unsupported_text_compression() which does e.g.:

    self.filesystem_client.create_file("{0}/fake.lz4".format(lz4_ym_partition_loc)[1:],
        "some test data")

Not only is this confusing, but it doesn't work when the test is run with FILESYSTEM_PREFIX not empty.

We should fix the filesystem_client classes to handle absolute and fully qualified paths so that each test doesn't have to worry about this.

Issue Links

• is related to: IMPALA-8344 Add support for running S3 tests with S3Guard
• relates to: IMPALA-7099 test_unsupported_text_compression fails: Expected one file per partition dir