Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-9004

TestCompressedFormats is broken for text files

    XMLWordPrintableJSON

Details

    • Test
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • Impala 4.0.0
    • None
    • None
    • ghx-label-4

    Description

      While working onĀ IMPALA-8950, we made a fix to TestCompressedFormats so that it actually checks the exit status of the hdfs dfs -cp command, turns out that this command has been silently failing whenever test_compressed_formats runs with file_format = text.

      For some reason, data load writes compressed text files with their corresponding file compression suffix, but for compressed seq/rc files, it does not:

      hdfs dfs -ls /test-warehouse/tinytable_seq_*
      Found 1 items
      -rwxr-xr-x   3 systest supergroup        325 2019-08-22 14:32 /test-warehouse/tinytable_seq_bzip/000000_0
      Found 1 items
      -rwxr-xr-x   3 systest supergroup        215 2019-08-22 14:32 /test-warehouse/tinytable_seq_def/000000_0
      Found 1 items
      -rwxr-xr-x   3 systest supergroup        260 2019-08-22 14:32 /test-warehouse/tinytable_seq_gzip/000000_0
      Found 1 items
      -rwxr-xr-x   3 systest supergroup        301 2019-08-22 14:32 /test-warehouse/tinytable_seq_record_bzip/000000_0
      Found 1 items
      -rwxr-xr-x   3 systest supergroup        209 2019-08-22 14:32 /test-warehouse/tinytable_seq_record_def/000000_0
      Found 1 items
      -rwxr-xr-x   3 systest supergroup        242 2019-08-22 14:32 /test-warehouse/tinytable_seq_record_gzip/000000_0
      Found 1 items
      -rwxr-xr-x   3 systest supergroup        233 2019-08-22 14:32 /test-warehouse/tinytable_seq_record_snap/000000_0
      Found 2 items
      -rwxr-xr-x   3 systest supergroup        243 2019-08-22 14:32 /test-warehouse/tinytable_seq_snap/000000_0
      
      hdfs dfs -ls /test-warehouse/tinytable_text_*
      Found 1 items
      -rwxr-xr-x   3 systest supergroup         59 2019-08-22 14:32 /test-warehouse/tinytable_text_bzip/000000_0.bz2
      Found 1 items
      -rwxr-xr-x   3 systest supergroup         28 2019-08-22 14:32 /test-warehouse/tinytable_text_def/000000_0.deflate
      Found 1 items
      -rwxr-xr-x   3 systest supergroup         40 2019-08-22 14:32 /test-warehouse/tinytable_text_gzip/000000_0.gz
      Found 2 items
      -rwxr-xr-x   3 systest supergroup         87 2019-08-22 14:32 /test-warehouse/tinytable_text_lzo/000000_0.lzo
      -rw-r--r--   3 systest supergroup          8 2019-08-22 14:42 /test-warehouse/tinytable_text_lzo/000000_0.lzo.index
      Found 1 items
      -rwxr-xr-x   3 systest supergroup         41 2019-08-22 14:32 /test-warehouse/tinytable_text_snap/000000_0.snappy

      Not sure if that is by design or not, but it is causing the tests to fail for all text files.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              stakiar Sahil Takiar
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: