Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-10653 Introduce Pluggable Shuffle Service Architecture
  3. FLINK-12735

Make shuffle environment implementation independent with IOManager

    XMLWordPrintableJSON

    Details

      Description

      The current creation of NetworkEnvironment is relying on IOManager from TaskManagerServices. In order not to rely on external specific components for implementing shuffle environment, and let the specific implementation creates internal components if required.

      The current abstract IOManager has two roles, one is for file channel management based on temp directories configuration, and the other is providing abstract methods for reading/writing files. We could further extract the file channel management as a separate internal class which could be reused for all the required components, like current BoundedBlockingSubpartition. To do so, the shuffle environment should also creates its internal channel manager to break dependency with IOManager.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                zjwang Zhijiang
                Reporter:
                zjwang Zhijiang
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 20m
                  20m