[IMPALA-14] Files with .gz extension reported as 'not supported'

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 0.3
• Fix Version/s: Impala 0.4
• Assignee: Alexander Behm
• Reporter: Henry Robinson
• Created: 07/Jan/13 21:13
• Updated: 14/May/15 22:32
• Resolved: 15/Jan/13 18:51

Description

See this block of code.

HdfsCompression compressionType =
            HdfsCompression.fromFileName(fileDescriptor.getFilePath());
        fileDescriptor.setCompression(compressionType);
        if (compressionType == HdfsCompression.LZO_INDEX) {
          // Skip index files, these are read by the LZO scanner directly.
          continue;
        }

        HdfsStorageDescriptor sd = partition.getInputFormatDescriptor();
        if (compressionType == HdfsCompression.LZO) {
          if (sd.getFileFormat() != HdfsFileFormat.LZO_TEXT) {
            throw new RuntimeException(
                "Compressed file not supported without compression input format: " + p);
          }
        } else if (compressionType != HdfsCompression.NONE) {
          throw new RuntimeException("Compressed file not supported: " + p);
        } else if (sd.getFileFormat() == HdfsFileFormat.LZO_TEXT) {
          throw new RuntimeException("Expected file with .lzo suffix: " + p);
        }

The compression type checks aren't correct: they shouldn't fail for .gz, and they arguably shouldn't be run for non-text formats.

Workaround for now is to rename files to .gz.