Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-2482

Adjust expected Parquet Footer size

    XMLWordPrintableJSON

Details

    • Task
    • Status: Resolved
    • Minor
    • Resolution: Won't Do
    • Impala 2.3.0
    • None
    • Backend

    Description

      The HdfsParquetScanner::FOOTER_SIZE is the field which holds the expected footer size. Since we do not know the exact footer size before hand, we do a disk read for at least this much and increase it if we do not reach the magic bytes "PAR1".

      This had been set to 100KB until now which would be an over estimation on the average case.

      This has to be brought down after some analysis on actual footer sizes of large data sets.

      Attachments

        Activity

          People

            sailesh Sailesh Mukil
            sailesh Sailesh Mukil
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: