Uploaded image for project: 'Parquet'
  1. Parquet
  2. PARQUET-409

InternalParquetRecordWriter doesn't use min/max row counts

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 1.8.1
    • None
    • parquet-mr

    Description

      PARQUET-99 added settings to control the min and max number of rows between size checks when flushing pages, and a setting to control whether to always use a static size (the min). The InternalParquetRecordWriter has similar checks that don't use those settings. We should determine if it should update it to use those settings or similar.

      Attachments

        Issue Links

          duplicates

          Improvement - An improvement or enhancement to an existing feature or task. PARQUET-869 Min/Max record counts for block size checks are not configurable

          • Major - Major loss of function.
          • Patch Available
          is related to

          Improvement - An improvement or enhancement to an existing feature or task. PARQUET-99 Large rows cause unnecessary OOM exceptions

          • Major - Major loss of function.
          • Resolved
          links to

          Web Link GitHub Pull Request #495

          Activity

            People

              Unassigned Unassigned
              rdblue Ryan Blue
              Votes:
              2 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated: