Uploaded image for project: 'Parquet'
  1. Parquet
  2. PARQUET-792

Skip the storage of repetition level and definition level for all-null column

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Minor
    • Resolution: Unresolved
    • None
    • None
    • parquet-mr
    • None

    Description

      I have a very sparse protobuf message in my project, with thousands of fields.
      In practise, most of the fields are all null values in one page.
      But the repetition level and definition level takes lots of storage space.
      Can parquet skip the storage of r level and d level for such all-null columns to save storage space?

      Attachments

        Activity

          People

            Unassigned Unassigned
            lirui-tx Li
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: