Uploaded image for project: 'Parquet'
  1. Parquet
  2. PARQUET-1006

ColumnChunkPageWriter uses only heap memory.

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 1.8.0, 1.12.0
    • None
    • parquet-mr
    • None

    Description

      After PARQUET-160 was resolved, ColumnChunkPageWriter started using ConcatenatingByteArrayCollector. There are all data is collected in the List of byte[], before writing the page. No way to use direct memory for allocating buffers. ByteBufferAllocator is present in the ColumnChunkPageWriter class, but never used.

      Using of java heap space in some cases can cause OOM exceptions or GC's overhead.
      ByteBufferAllocator should be used in the ConcatenatingByteArrayCollector or OutputStream classes.

      Attachments

        Issue Links

          Is contained by

          Bug - A problem which impairs or prevents the functions of the product. DRILL-7825 Error: SYSTEM ERROR: RuntimeException: Unknown logical type <LogicalType UUID:UUIDType()>

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. DRILL-7906 Replace ParquetColumnChunkPageWriter with original Parquet class

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open
          is related to

          Improvement - An improvement or enhancement to an existing feature or task. PARQUET-1771 Support configurable for DirectByteBufferAllocator from Hadoop Configuration

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. PARQUET-77 Improvements in ByteBuffer read path

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. PARQUET-160 Simplify CapacityByteArrayOutputStream

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          relates to

          Bug - A problem which impairs or prevents the functions of the product. DRILL-5544 Out of heap running CTAS against text delimited

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              vitalii Vitalii Diravka
              vitalii Vitalii Diravka
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated: