Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-13540

DFSStripedInputStream should only allocate new buffers when reading

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0
    • Fix Version/s: 3.2.0, 3.1.1, 3.0.3
    • Component/s: None
    • Labels:
      None
    • Target Version/s:

      Description

      This was found in the same scenario where HDFS-13539 is caught.

      There are 2 OOM that looks interesting:

      FSDataInputStream#close error:
      OutOfMemoryError: Direct buffer memoryjava.lang.OutOfMemoryError: Direct buffer memory
              at java.nio.Bits.reserveMemory(Bits.java:694)
              at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
              at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
              at org.apache.hadoop.io.ElasticByteBufferPool.getBuffer(ElasticByteBufferPool.java:95)
              at org.apache.hadoop.hdfs.DFSStripedInputStream.resetCurStripeBuffer(DFSStripedInputStream.java:118)
              at org.apache.hadoop.hdfs.DFSStripedInputStream.closeCurrentBlockReaders(DFSStripedInputStream.java:205)
              at org.apache.hadoop.hdfs.DFSInputStream.close(DFSInputStream.java:672)
              at org.apache.hadoop.hdfs.DFSStripedInputStream.close(DFSStripedInputStream.java:181)
              at java.io.FilterInputStream.close(FilterInputStream.java:181)
      

      and

      org/apache/hadoop/fs/FSDataInputStream#unbuffer failed: error:
      OutOfMemoryError: Direct buffer memoryjava.lang.OutOfMemoryError: Direct buffer memory
              at java.nio.Bits.reserveMemory(Bits.java:694)
              at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
              at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
              at org.apache.hadoop.io.ElasticByteBufferPool.getBuffer(ElasticByteBufferPool.java:95)
              at org.apache.hadoop.hdfs.DFSStripedInputStream.resetCurStripeBuffer(DFSStripedInputStream.java:118)
              at org.apache.hadoop.hdfs.DFSStripedInputStream.closeCurrentBlockReaders(DFSStripedInputStream.java:205)
              at org.apache.hadoop.hdfs.DFSInputStream.unbuffer(DFSInputStream.java:1782)
              at org.apache.hadoop.fs.StreamCapabilitiesPolicy.unbuffer(StreamCapabilitiesPolicy.java:48)
              at org.apache.hadoop.fs.FSDataInputStream.unbuffer(FSDataInputStream.java:230)
      

      As the stack trace goes, resetCurStripeBuffer will get buffer from the buffer pool. We could save the cost of doing so if it's not for a read (e.g. close, unbuffer etc.)

        Attachments

        1. HDFS-13540.06.patch
          5 kB
          Xiao Chen
        2. HDFS-13540.05.patch
          5 kB
          Xiao Chen
        3. HDFS-13540.04.patch
          4 kB
          Xiao Chen
        4. HDFS-13540.03.patch
          3 kB
          Xiao Chen
        5. HDFS-13540.02.patch
          3 kB
          Xiao Chen
        6. HDFS-13540.01.patch
          3 kB
          Xiao Chen

          Activity

            People

            • Assignee:
              xiaochen Xiao Chen
              Reporter:
              xiaochen Xiao Chen
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: