Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-18915

FIXED_PATH(dummy Hadoop Path) with WriterImpl may cause ORC writer OOM

    XMLWordPrintableJSON

    Details

      Description

      1. OrcBulkWriterFactory
        @Override
        public BulkWriter<T> create(FSDataOutputStream out) throws IOException {
           OrcFile.WriterOptions opts = getWriterOptions();
           opts.physicalWriter(new PhysicalWriterImpl(out, opts));
        
           return new OrcBulkWriter<>(vectorizer, new WriterImpl(null, FIXED_PATH, opts));
        }

         

      # MemoryManagerImpl

      // 
      public void addWriter(Path path, long requestedAllocation,
                                  Callback callback) throws IOException {
        checkOwner();
        WriterInfo oldVal = writerList.get(path);
        // this should always be null, but we handle the case where the memory
        // manager wasn't told that a writer wasn't still in use and the task
        // starts writing to the same path.
        if (oldVal == null) {
          oldVal = new WriterInfo(requestedAllocation, callback);
          writerList.put(path, oldVal);
          totalAllocation += requestedAllocation;
        } else {
          // handle a new writer that is writing to the same path
          totalAllocation += requestedAllocation - oldVal.allocation;
          oldVal.allocation = requestedAllocation;
          oldVal.callback = callback;
        }
        updateScale(true);
      }
      

      SinkTask may have multi BulkWriter create, FIXED_PATH will cause overlay the last writer callback;Last writer's WriterImpl#checkMemory will never called;

       

       

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                gaoyunhaii Yun Gao
                Reporter:
                wei.wei wei
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: