[IMPALA-9802] TestCompressedFormats.test_compressed_formats fails in HDFS copy

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 4.0.0
• Fix Version/s: Impala 4.0.0
• Labels: broken-build
• Epic Color: ghx-label-7
• Assignee: Xiaomeng Zhang
• Reporter: Quanlong Huang
• Created: 29/May/20 14:39
• Updated: 09/Jun/20 17:09
• Resolved: 09/Jun/20 17:09

Description

See this fails in rc and seq file formats:

query_test.test_compressed_formats.TestCompressedFormats.test_compressed_formats[compression_format: ('.bz2', 'bzip') | file_format: rc]
query_test.test_compressed_formats.TestCompressedFormats.test_compressed_formats[compression_format: ('.deflate', 'def') | file_format: rc]
query_test.test_compressed_formats.TestCompressedFormats.test_compressed_formats[compression_format: ('.gz', 'gzip') | file_format: rc]
query_test.test_compressed_formats.TestCompressedFormats.test_compressed_formats[compression_format: ('.snappy', 'snap') | file_format: rc]
query_test.test_compressed_formats.TestCompressedFormats.test_compressed_formats[compression_format: ('.snappy', 'snap') | file_format: seq]
query_test.test_compressed_formats.TestCompressedFormats.test_compressed_formats[compression_format: ('.gz', 'gzip') | file_format: seq]
query_test.test_compressed_formats.TestCompressedFormats.test_compressed_formats[compression_format: ('.deflate', 'def') | file_format: seq]
query_test.test_compressed_formats.TestCompressedFormats.test_compressed_formats[compression_format: ('.bz2', 'bzip') | file_format: seq]

One of the stacktrace:

query_test/test_compressed_formats.py:83: in test_compressed_formats
    'tinytable', db_suffix, suffix, '000000_0', extension)
query_test/test_compressed_formats.py:114: in _copy_and_query_compressed_file
    self.filesystem_client.copy(src_file, dest_file, overwrite=True)
/data/jenkins/workspace/impala-cdpd-master-exhaustive/repos/Impala/tests/util/hdfs_util.py:79: in copy
    self.hdfs_filesystem_client.copy(src, dst, overwrite)
/data/jenkins/workspace/impala-cdpd-master-exhaustive/repos/Impala/tests/util/hdfs_util.py:241: in copy
    '{0} copy failed: '.format(self.filesystem_type) + stderr + "; " + stdout
E   AssertionError: HDFS copy failed: cp: `/test-warehouse/managed/tinytable_snap_copy/000000_0.snappy': No such file or directory: `hdfs://localhost:20500/test-warehouse/managed/tinytable_snap_copy/000000_0.snappy'
E   ; 

Issue Links

• is related to: IMPALA-9673 Tests expecting results to be in test-warehouse/managed but find test-warehouse