Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-5287

Add a test for skip.header.line.count on compressed files

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: Impala 2.9.0
    • Fix Version/s: Impala 2.9.0
    • Component/s: Backend
    • Labels:
      None

      Description

      Before the fix for IMPALA-3905 was merged, the HDFS text scanner initialized the decompressor after finding the first row. This was wrong, but not an issue for normal compressed tables, since for those we only issue a single scan range, ant therefore can skip searching for the first newline character.

      However, this broke skipping header lines at the beginning of compressed files. We should add a test for skip.header.line.count on compressed files to prevent a regression in the future.

        Issue Links

          Activity

          Hide
          lv Lars Volker added a comment -

          IMPALA-5287: Test skip.header.line.count on gzip

          This change fixed IMPALA-4873 by adding the capability to supply a dict
          'test_file_vars' to run_test_case(). Keys in this dict will be replaced
          with their values inside test queries before they are executed.

          Change-Id: Ie3f3c29a42501cfb2751f7ad0af166eb88f63b70
          Reviewed-on: http://gerrit.cloudera.org:8080/6817
          Reviewed-by: Michael Brown <mikeb@cloudera.com>
          Tested-by: Impala Public Jenkins

          Show
          lv Lars Volker added a comment - IMPALA-5287 : Test skip.header.line.count on gzip This change fixed IMPALA-4873 by adding the capability to supply a dict 'test_file_vars' to run_test_case(). Keys in this dict will be replaced with their values inside test queries before they are executed. Change-Id: Ie3f3c29a42501cfb2751f7ad0af166eb88f63b70 Reviewed-on: http://gerrit.cloudera.org:8080/6817 Reviewed-by: Michael Brown <mikeb@cloudera.com> Tested-by: Impala Public Jenkins

            People

            • Assignee:
              lv Lars Volker
              Reporter:
              lv Lars Volker
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development