[SPARK-21517] Fetch local data via block manager cause oom

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.6.1, 2.1.0
• Fix Version/s: 2.3.0
• Component/s: Block Manager, Spark Core
• Assignee: zhoukang
• Reporter: zhoukang
• Created: 24/Jul/17 08:13
• Updated: 20/Sep/18 17:23
• Resolved: 26/Jul/17 00:59

Description

In our production cluster,oom happens when NettyBlockRpcServer receive OpenBlocks message.The reason we observed is below:
When BlockManagerManagedBuffer call ChunkedByteBuffer#toNetty, it will use Unpooled.wrappedBuffer(ByteBuffer... buffers) which use default maxNumComponents=16 in low-level CompositeByteBuf.When our component's number is bigger than 16, it will execute during buffer copy.

private void consolidateIfNeeded() {
        int numComponents = this.components.size();
        if(numComponents > this.maxNumComponents) {
            int capacity = ((CompositeByteBuf.Component)this.components.get(numComponents - 1)).endOffset;
            ByteBuf consolidated = this.allocBuffer(capacity);

            for(int c = 0; c < numComponents; ++c) {
                CompositeByteBuf.Component c1 = (CompositeByteBuf.Component)this.components.get(c);
                ByteBuf b = c1.buf;
                consolidated.writeBytes(b);
                c1.freeIfNecessary();
            }

            CompositeByteBuf.Component var7 = new CompositeByteBuf.Component(consolidated);
            var7.endOffset = var7.length;
            this.components.clear();
            this.components.add(var7);
        }

    }

in CompositeByteBuf which will consume some memory during buffer copy.

Issue Links

• is related to: SPARK-25055 MessageWithHeader transfer ByteBuffer from Netty's CompositeByteBuf many times
• links to: [Github] Pull Request #18723 (caneGuy)