Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-10653 Introduce Pluggable Shuffle Service Architecture
  3. FLINK-11776

Refactor to simplify the process of scheduleOrUpdateConsumers

    XMLWordPrintableJSON

    Details

      Description

      Based on the work of FLINK-11417, the process of Execution#scheduleOrUpdateConsumers do not need to consider the race condition with schedule process, so we can refactor to make it easy to handle.

      • The concurrent data structure for cacheing partial input channel descriptor can be replaced by common list.
      • If the consumer is in CREATED state, we only need to schedule it and the partition info would be known during deployment.
      • If the consumer is in SCHEDULED state, we need do nothing.
      • If the consumer is in RUNNING state, we can send partition info immediately.
      • If the consumer is in DEPLOYING state, we can cache the partition info in order to send them in batch after consumer switching to RUNNING state.
      • PartialInputChannelDeploymentDescriptor is not needed any more, we can cache partition info directly.

      This refactoring is also a preparation work for future introducing ShuffleMaster in FLINK-11391.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                zjwang Zhijiang
                Reporter:
                zjwang Zhijiang
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 20m
                  20m