Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-14551 Unaligned checkpoints
  3. FLINK-14396

Implement rudimentary non-blocking network output

    XMLWordPrintableJSON

    Details

      Description

      Considering the mailbox model and unaligned checkpoints requirements in future, task network output should be non-blocking. In other words, as long as output is available, it should never block for a subsequent/future single record write.

      In the first version, we only implement the non-blocking output for the most regular case, and do not solve the following cases which still keep the previous behavior.

      • Big record which might span multiple buffers
      • Flatmap-like operators which might emit multiple records in every process
      • Broadcast watermark which might request multiple buffers at a time

      The solution is providing the RecordWriter#isAvailable method and respective LocalBufferPool#isAvailable for judging the output beforehand. As long as there is at-least one available buffer in LocalBufferPool, the RecordWriter is available for network output in most cases.  This doesn’t include runtime handling of this non-blocking and availability behavior in StreamInputProcessor

      Note: It requires the minimum number of buffers in output LocalBufferPool adjusting to (numberOfSubpartitions + 1) and also adjusting the monitor of backpressure future.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                zjwang Zhijiang
                Reporter:
                zjwang Zhijiang
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 20m
                  20m