Uploaded image for project: 'Parquet'
  1. Parquet
  2. PARQUET-852

Slowly ramp up sizes of byte[] in ByteBasedBitPackingEncoder

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • None
    • 1.10.0
    • None
    • None

    Description

      The current allocation policy for ByteBasedBitPackingEncoder is to allocate 64KB * #bits up-front. As similarly observed in PARQUET-580, this can lead to significant memory overheads for high-fanout scenarios (many columns and/or open files, in my case using BooleanPlainValuesWriter).

      As done in PARQUET-585, I'll follow up with a PR that starts with a smaller buffer and works its way up to a max.

      Attachments

        Issue Links

          links to

          Web Link GitHub Pull Request #467

          Activity

            People

              Unassigned Unassigned
              jjenkins John Jenkins
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: