Uploaded image for project: 'Parquet'
  1. Parquet
  2. PARQUET-1006

ColumnChunkPageWriter uses only heap memory.

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.8.0
    • Fix Version/s: None
    • Component/s: parquet-mr
    • Labels:
      None

      Description

      After PARQUET-160 was resolved, ColumnChunkPageWriter started using ConcatenatingByteArrayCollector. There are all data is collected in the List of byte[], before writing the page. No way to use direct memory for allocating buffers. ByteBufferAllocator is present in the ColumnChunkPageWriter class, but never used.

      Using of java heap space in some cases can cause OOM exceptions or GC's overhead.
      ByteBufferAllocator should be used in the ConcatenatingByteArrayCollector or OutputStream classes.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                vitalii Vitalii Diravka
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated: