Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-7282

Credit-based Network Flow Control

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • None
    • Runtime / Network
    • None

    Description

      This is a part of work for network stack improvements proposed in StephanEwen 's FLIP

      Backpressure currently happens very naturally through the TCP network connections and the bounded buffering capacity. The downsides are :

      • All channels multiplexed into the same TCP connection stall together, as soon as one channel has backpressure.
      • Under backpressure, connections can not transport checkpoint barriers.

      This flink-managed flow control is similar to the window-based advertisement mechanism in TCP. The basic approaches are the following:

      • Each RemoteInputChannel has fixed exclusive buffers as initial credits, and SingleInputGate has a fixed buffer pool for managing floating buffers for all RemoteInputChannels.
      • RemoteInputChannel as receiver notifies the current available credits to the sender side.
      • Senders must never send buffers without credit, that means all the buffers sent must be accepted by receivers so no buffers accumulated on the network wire.
      • Senders also send the current size of backlog that indicates how many buffers are available on the sender side. The receivers use this information to decide how to request floating buffers from the fixed buffer pool.

      To avoid immediate commits affecting master branch, it will be implemented into a separate feature branch.

      Attachments

        There are no Sub-Tasks for this issue.

        Activity

          People

            zjwang Zhijiang
            zjwang Zhijiang
            Votes:
            1 Vote for this issue
            Watchers:
            19 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 1h 40m
                1h 40m