Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-4988

Add query option to control filtering based on parquet::Statistics

    Details

      Description

      In order to deal with broken Parquet files and potentially hidden bugs in our code, we should have a query option to toggle parquet::Statistics based filtering.

        Issue Links

          Activity

          Hide
          lv Lars Volker added a comment -

          IMPALA-4988: Add query option read_parquet_statistics

          This change adds a query option to disable reading Parquet statistics.
          It provides a workaround when dealing with files that have corrupt
          parquet statistics.

          Note that Impala handles Parquet files affected by PARQUET-251 correctly
          by ignoring statistics for anything but plain numeric types. This query
          option is supposed to help with files affected by unknown or errors or
          by errors that are yet to be made.

          Change-Id: I427f7fde40d0f4b703751e40f3c2109a850643f7
          Reviewed-on: http://gerrit.cloudera.org:8080/7001
          Reviewed-by: Dan Hecht <dhecht@cloudera.com>
          Tested-by: Impala Public Jenkins

          Show
          lv Lars Volker added a comment - IMPALA-4988 : Add query option read_parquet_statistics This change adds a query option to disable reading Parquet statistics. It provides a workaround when dealing with files that have corrupt parquet statistics. Note that Impala handles Parquet files affected by PARQUET-251 correctly by ignoring statistics for anything but plain numeric types. This query option is supposed to help with files affected by unknown or errors or by errors that are yet to be made. Change-Id: I427f7fde40d0f4b703751e40f3c2109a850643f7 Reviewed-on: http://gerrit.cloudera.org:8080/7001 Reviewed-by: Dan Hecht <dhecht@cloudera.com> Tested-by: Impala Public Jenkins

            People

            • Assignee:
              lv Lars Volker
              Reporter:
              lv Lars Volker
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development