Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-21517

Fetch local data via block manager cause oom

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.6.1, 2.1.0
    • Fix Version/s: 2.3.0
    • Component/s: Block Manager, Spark Core
    • Labels:
      None

      Description

      In our production cluster,oom happens when NettyBlockRpcServer receive OpenBlocks message.The reason we observed is below:
      When BlockManagerManagedBuffer call ChunkedByteBuffer#toNetty, it will use Unpooled.wrappedBuffer(ByteBuffer... buffers) which use default maxNumComponents=16 in low-level CompositeByteBuf.When our component's number is bigger than 16, it will execute during buffer copy.

      private void consolidateIfNeeded() {
              int numComponents = this.components.size();
              if(numComponents > this.maxNumComponents) {
                  int capacity = ((CompositeByteBuf.Component)this.components.get(numComponents - 1)).endOffset;
                  ByteBuf consolidated = this.allocBuffer(capacity);
      
                  for(int c = 0; c < numComponents; ++c) {
                      CompositeByteBuf.Component c1 = (CompositeByteBuf.Component)this.components.get(c);
                      ByteBuf b = c1.buf;
                      consolidated.writeBytes(b);
                      c1.freeIfNecessary();
                  }
      
                  CompositeByteBuf.Component var7 = new CompositeByteBuf.Component(consolidated);
                  var7.endOffset = var7.length;
                  this.components.clear();
                  this.components.add(var7);
              }
      
          }
      

      in CompositeByteBuf which will consume some memory during buffer copy.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                caneGuy zhoukang
                Reporter:
                cane zhoukang
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: