[FLINK-18915] FIXED_PATH(dummy Hadoop Path) with WriterImpl may cause ORC writer OOM

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.11.0, 1.11.1
• Fix Version/s: 1.11.3, 1.12.0
• Component/s: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
• Labels: pull-request-available
• Assignee: Yun Gao
• Reporter: wei
• Created: 13/Aug/20 04:52
• Updated: 17/Oct/20 18:20
• Resolved: 16/Oct/20 17:59

Description

1. OrcBulkWriterFactory

   @Override
   public BulkWriter<T> create(FSDataOutputStream out) throws IOException {
      OrcFile.WriterOptions opts = getWriterOptions();
      opts.physicalWriter(new PhysicalWriterImpl(out, opts));
   
      return new OrcBulkWriter<>(vectorizer, new WriterImpl(null, FIXED_PATH, opts));
   }

    

# MemoryManagerImpl

// 
public void addWriter(Path path, long requestedAllocation,
                            Callback callback) throws IOException {
  checkOwner();
  WriterInfo oldVal = writerList.get(path);
  // this should always be null, but we handle the case where the memory
  // manager wasn't told that a writer wasn't still in use and the task
  // starts writing to the same path.
  if (oldVal == null) {
    oldVal = new WriterInfo(requestedAllocation, callback);
    writerList.put(path, oldVal);
    totalAllocation += requestedAllocation;
  } else {
    // handle a new writer that is writing to the same path
    totalAllocation += requestedAllocation - oldVal.allocation;
    oldVal.allocation = requestedAllocation;
    oldVal.callback = callback;
  }
  updateScale(true);
}

SinkTask may have multi BulkWriter create, FIXED_PATH will cause overlay the last writer callback；Last writer's WriterImpl#checkMemory will never called;

 

 

Issue Links

• links to: GitHub Pull Request #13598