Uploaded image for project: 'Parquet'
  1. Parquet
  2. PARQUET-1320

Fast clean unused direct memory when decompress

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: parquet-mr

      Description

      When use NonBlockedDecompressorStream which call:

      SnappyDecompressor.setInput

      public synchronized void setInput(byte[] buffer, int off, int len) {
       SnappyUtil
      
      public synchronized void setInput(byte[] buffer, int off, int len) {
       SnappyUtil.validateBuffer(buffer, off, len);
      
       if (inputBuffer.capacity() - inputBuffer.position() < len) {
       ByteBuffer newBuffer = ByteBuffer.allocateDirect(inputBuffer.position() + len);
       inputBuffer.rewind();
       newBuffer.put(inputBuffer);
       inputBuffer = newBuffer; 
       } else {
       inputBuffer.limit(inputBuffer.position() + len);
       }
       inputBuffer.put(buffer, off, len);
      }
      
      .validateBuffer(buffer, off, len);
      
       if (inputBuffer.capacity() - inputBuffer.position() < len) {
       ByteBuffer newBuffer = ByteBuffer.allocateDirect(inputBuffer.position() + len);
       inputBuffer.rewind();
       newBuffer.put(inputBuffer);
       inputBuffer = newBuffer; 
       } else {
       inputBuffer.limit(inputBuffer.position() + len);
       }
       inputBuffer.put(buffer, off, len);
      }
      
      

      If we do not get any full gc for old gen.we may failed by off-heap memory leak
       

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                cane zhoukang
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated: