[HDFS-13540] DFSStripedInputStream should only allocate new buffers when reading

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.2.0, 3.1.1, 3.0.3
• Target Version/s: 3.0.3
• Assignee: Xiao Chen
• Reporter: Xiao Chen
• Created: 08/May/18 22:50
• Updated: 24/May/18 04:18
• Resolved: 23/May/18 12:08

Description

This was found in the same scenario where HDFS-13539 is caught.

There are 2 OOM that looks interesting:

FSDataInputStream#close error:
OutOfMemoryError: Direct buffer memoryjava.lang.OutOfMemoryError: Direct buffer memory
        at java.nio.Bits.reserveMemory(Bits.java:694)
        at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
        at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
        at org.apache.hadoop.io.ElasticByteBufferPool.getBuffer(ElasticByteBufferPool.java:95)
        at org.apache.hadoop.hdfs.DFSStripedInputStream.resetCurStripeBuffer(DFSStripedInputStream.java:118)
        at org.apache.hadoop.hdfs.DFSStripedInputStream.closeCurrentBlockReaders(DFSStripedInputStream.java:205)
        at org.apache.hadoop.hdfs.DFSInputStream.close(DFSInputStream.java:672)
        at org.apache.hadoop.hdfs.DFSStripedInputStream.close(DFSStripedInputStream.java:181)
        at java.io.FilterInputStream.close(FilterInputStream.java:181)

and

org/apache/hadoop/fs/FSDataInputStream#unbuffer failed: error:
OutOfMemoryError: Direct buffer memoryjava.lang.OutOfMemoryError: Direct buffer memory
        at java.nio.Bits.reserveMemory(Bits.java:694)
        at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
        at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
        at org.apache.hadoop.io.ElasticByteBufferPool.getBuffer(ElasticByteBufferPool.java:95)
        at org.apache.hadoop.hdfs.DFSStripedInputStream.resetCurStripeBuffer(DFSStripedInputStream.java:118)
        at org.apache.hadoop.hdfs.DFSStripedInputStream.closeCurrentBlockReaders(DFSStripedInputStream.java:205)
        at org.apache.hadoop.hdfs.DFSInputStream.unbuffer(DFSInputStream.java:1782)
        at org.apache.hadoop.fs.StreamCapabilitiesPolicy.unbuffer(StreamCapabilitiesPolicy.java:48)
        at org.apache.hadoop.fs.FSDataInputStream.unbuffer(FSDataInputStream.java:230)

As the stack trace goes, resetCurStripeBuffer will get buffer from the buffer pool. We could save the cost of doing so if it's not for a read (e.g. close, unbuffer etc.)